2023-11-01T04:01:37.869 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-01T04:01:37.870 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312 --name benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi -- /home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312/orig.config.yaml 2023-11-01T04:01:37.914 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-01T04:01:38.000 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312 branch: DBG_maintain_prefix_itr_2 description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} email: null first_in_suite: false job_id: '7442312' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: DBG_maintain_prefix_itr_2 ceph: conf: global: mon election default strategy: 1 osd debug inject dispatch delay duration: 0.1 osd debug inject dispatch delay probability: 0.1 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from sha1: 420cd129397301b1521ceb74e1bdbd9b855d2189 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 420cd129397301b1521ceb74e1bdbd9b855d2189 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: DBG_maintain_prefix_itr_2 sha1: 420cd129397301b1521ceb74e1bdbd9b855d2189 owner: scheduled_benhanokh@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 9009 sha1: 420cd129397301b1521ceb74e1bdbd9b855d2189 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: DBG_maintain_prefix_itr_2 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 420cd129397301b1521ceb74e1bdbd9b855d2189 targets: smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOAGLJhjDq8mvVkTG8Hw5tTgR0UZXjowuOsiYalT+QNbCXSyYSvs8LG7L/jZHCvB0sTKLuOqXRMsQl+LlBrIlOM= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKZi16MWKJndbPT3R1FdevTMb+dXFqXxHmJE/P6kQ+Gh9wecRp0CFuX/SllIWmHsP+6ZRYdXM51iHWKE79Ov4VI= smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG6klwHOgaJO69D4LNOd3TcLmJkkJ3EdPRk+R27RAzf40b49HV2S9P8kZbO/WVyXSV3JNaS4kVTEr6Fr0WSh2BI= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCRm8aIzpGl2fqDE99OAe/4/JqhCP3sSy1lyMbrSiFzkg5qUKVPKtmAiues2krHh9LOorVPXIHYKqqHJZs9ZqpM= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-10-31_16:43:39 tube: smithi user: benhanokh verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-11-01T04:01:38.001 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa; will attempt to use it 2023-11-01T04:01:38.002 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks 2023-11-01T04:01:38.002 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-01T04:01:38.004 INFO:teuthology.task.internal:Checking packages... 2023-11-01T04:01:38.027 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '420cd129397301b1521ceb74e1bdbd9b855d2189' 2023-11-01T04:01:38.027 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-01T04:01:38.028 INFO:teuthology.packaging:ref: None 2023-11-01T04:01:38.028 INFO:teuthology.packaging:tag: None 2023-11-01T04:01:38.029 INFO:teuthology.packaging:branch: DBG_maintain_prefix_itr_2 2023-11-01T04:01:38.029 INFO:teuthology.packaging:sha1: 420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-01T04:01:38.029 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=DBG_maintain_prefix_itr_2 2023-11-01T04:01:38.176 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6776-g420cd129-1jammy 2023-11-01T04:01:38.177 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-01T04:01:38.183 INFO:teuthology.task.internal:no buildpackages task found 2023-11-01T04:01:38.184 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-01T04:01:38.189 INFO:teuthology.task.internal:Saving configuration 2023-11-01T04:01:38.205 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-01T04:01:38.211 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-01T04:01:38.236 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-01 03:56:01.288015', 'locked_by': 'scheduled_benhanokh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ4WGH74TJpuXGs9+bNx1urdlsn74qFQP+68J0gjiGXb'} 2023-11-01T04:01:38.259 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-01 03:56:01.289629', 'locked_by': 'scheduled_benhanokh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILD+W5CxvwxSQJkMP6tUPquxpxBH7L8ZxlcQkzWmbTXt'} 2023-11-01T04:01:38.281 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-01 03:56:01.293004', 'locked_by': 'scheduled_benhanokh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFJhSHmxPks6sekS7keUGcPo1mszgBrLZNl9cpXF0sES'} 2023-11-01T04:01:38.305 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-01 03:56:01.291421', 'locked_by': 'scheduled_benhanokh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOKcUheaTjo6B4/si+c9PIuEZAIfbR/SzOEq+cUdWIxj'} 2023-11-01T04:01:38.305 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-01T04:01:38.312 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2023-11-01T04:01:38.312 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2023-11-01T04:01:38.313 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2023-11-01T04:01:38.313 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2023-11-01T04:01:38.313 INFO:teuthology.run_tasks:Running task console_log... 2023-11-01T04:01:38.451 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f510b7ffaf0>, signals=[15]) 2023-11-01T04:01:38.452 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-01T04:01:38.458 INFO:teuthology.task.internal:Opening connections... 2023-11-01T04:01:38.459 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2023-11-01T04:01:38.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:01:38.539 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2023-11-01T04:01:38.540 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:01:38.618 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2023-11-01T04:01:38.619 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:01:38.693 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2023-11-01T04:01:38.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:01:38.767 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-01T04:01:38.774 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2023-11-01T04:01:38.780 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2023-11-01T04:01:38.781 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2023-11-01T04:01:38.830 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-11-01T04:01:38.831 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="Ubuntu" 2023-11-01T04:01:38.831 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="22.04" 2023-11-01T04:01:38.831 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-11-01T04:01:38.831 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_CODENAME=jammy 2023-11-01T04:01:38.832 INFO:teuthology.orchestra.run.smithi153.stdout:ID=ubuntu 2023-11-01T04:01:38.832 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE=debian 2023-11-01T04:01:38.832 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-01T04:01:38.832 INFO:teuthology.orchestra.run.smithi153.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-01T04:01:38.832 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-01T04:01:38.833 INFO:teuthology.orchestra.run.smithi153.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-01T04:01:38.833 INFO:teuthology.orchestra.run.smithi153.stdout:UBUNTU_CODENAME=jammy 2023-11-01T04:01:38.834 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2023-11-01T04:01:38.858 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2023-11-01T04:01:38.865 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2023-11-01T04:01:38.865 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2023-11-01T04:01:38.913 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-11-01T04:01:38.913 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="Ubuntu" 2023-11-01T04:01:38.914 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="22.04" 2023-11-01T04:01:38.914 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-11-01T04:01:38.914 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_CODENAME=jammy 2023-11-01T04:01:38.914 INFO:teuthology.orchestra.run.smithi173.stdout:ID=ubuntu 2023-11-01T04:01:38.915 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE=debian 2023-11-01T04:01:38.915 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-01T04:01:38.915 INFO:teuthology.orchestra.run.smithi173.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-01T04:01:38.915 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-01T04:01:38.915 INFO:teuthology.orchestra.run.smithi173.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-01T04:01:38.916 INFO:teuthology.orchestra.run.smithi173.stdout:UBUNTU_CODENAME=jammy 2023-11-01T04:01:38.916 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2023-11-01T04:01:38.941 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2023-11-01T04:01:38.948 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2023-11-01T04:01:38.948 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2023-11-01T04:01:38.995 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-11-01T04:01:38.995 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="Ubuntu" 2023-11-01T04:01:38.995 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="22.04" 2023-11-01T04:01:38.996 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-11-01T04:01:38.996 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_CODENAME=jammy 2023-11-01T04:01:38.996 INFO:teuthology.orchestra.run.smithi181.stdout:ID=ubuntu 2023-11-01T04:01:38.996 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE=debian 2023-11-01T04:01:38.997 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-01T04:01:38.997 INFO:teuthology.orchestra.run.smithi181.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-01T04:01:38.997 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-01T04:01:38.997 INFO:teuthology.orchestra.run.smithi181.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-01T04:01:38.997 INFO:teuthology.orchestra.run.smithi181.stdout:UBUNTU_CODENAME=jammy 2023-11-01T04:01:38.998 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2023-11-01T04:01:39.025 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2023-11-01T04:01:39.031 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2023-11-01T04:01:39.032 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2023-11-01T04:01:39.079 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-11-01T04:01:39.079 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="Ubuntu" 2023-11-01T04:01:39.079 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="22.04" 2023-11-01T04:01:39.080 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-11-01T04:01:39.080 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_CODENAME=jammy 2023-11-01T04:01:39.080 INFO:teuthology.orchestra.run.smithi190.stdout:ID=ubuntu 2023-11-01T04:01:39.080 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE=debian 2023-11-01T04:01:39.081 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://www.ubuntu.com/" 2023-11-01T04:01:39.081 INFO:teuthology.orchestra.run.smithi190.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-11-01T04:01:39.081 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-11-01T04:01:39.081 INFO:teuthology.orchestra.run.smithi190.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-11-01T04:01:39.082 INFO:teuthology.orchestra.run.smithi190.stdout:UBUNTU_CODENAME=jammy 2023-11-01T04:01:39.082 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2023-11-01T04:01:39.108 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-01T04:01:39.118 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-01T04:01:39.123 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-01T04:01:39.124 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2023-11-01T04:01:39.126 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2023-11-01T04:01:39.130 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2023-11-01T04:01:39.132 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2023-11-01T04:01:39.137 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-01T04:01:39.143 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-01T04:01:39.143 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2023-11-01T04:01:39.175 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2023-11-01T04:01:39.178 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2023-11-01T04:01:39.182 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2023-11-01T04:01:39.198 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-01T04:01:39.298 INFO:teuthology.run_tasks:Running task kernel... 2023-11-01T04:01:39.314 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-01T04:01:39.315 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-01T04:01:39.315 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-01T04:01:39.316 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-01T04:01:39.317 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-01T04:01:39.317 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-01T04:01:39.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:01:39.322 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2023-11-01T04:01:39.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:01:39.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:01:39.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:01:39.325 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2023-11-01T04:01:39.326 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2023-11-01T04:01:39.326 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2023-11-01T04:01:39.370 INFO:teuthology.orchestra.run.smithi190.stdout:5.15.0-56-generic 2023-11-01T04:01:39.371 INFO:teuthology.task.kernel:Running kernel on smithi190: 5.15.0-56-generic 2023-11-01T04:01:39.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get clean 2023-11-01T04:01:39.373 INFO:teuthology.orchestra.run.smithi181.stdout:5.15.0-56-generic 2023-11-01T04:01:39.373 INFO:teuthology.task.kernel:Running kernel on smithi181: 5.15.0-56-generic 2023-11-01T04:01:39.374 DEBUG:teuthology.orchestra.run.smithi181:> sudo apt-get clean 2023-11-01T04:01:39.375 INFO:teuthology.orchestra.run.smithi153.stdout:5.15.0-56-generic 2023-11-01T04:01:39.375 INFO:teuthology.orchestra.run.smithi173.stdout:5.15.0-56-generic 2023-11-01T04:01:39.376 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.15.0-56-generic 2023-11-01T04:01:39.376 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get clean 2023-11-01T04:01:39.376 INFO:teuthology.task.kernel:Running kernel on smithi173: 5.15.0-56-generic 2023-11-01T04:01:39.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo apt-get clean 2023-11-01T04:01:39.502 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get update 2023-11-01T04:01:39.504 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2023-11-01T04:01:39.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo apt-get update 2023-11-01T04:01:39.509 DEBUG:teuthology.orchestra.run.smithi181:> sudo apt-get update 2023-11-01T04:01:39.660 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-01T04:01:39.661 INFO:teuthology.orchestra.run.smithi181.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-11-01T04:01:39.664 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-01T04:01:39.678 INFO:teuthology.orchestra.run.smithi173.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-11-01T04:01:39.679 INFO:teuthology.orchestra.run.smithi190.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-11-01T04:01:39.743 INFO:teuthology.orchestra.run.smithi173.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-11-01T04:01:39.752 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-11-01T04:01:39.770 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-01T04:01:39.772 INFO:teuthology.orchestra.run.smithi173.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-11-01T04:01:39.773 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-11-01T04:01:39.789 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-01T04:01:39.797 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-11-01T04:01:39.852 INFO:teuthology.orchestra.run.smithi181.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-11-01T04:01:39.852 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-11-01T04:01:39.889 INFO:teuthology.orchestra.run.smithi181.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [940 kB] 2023-11-01T04:01:39.905 INFO:teuthology.orchestra.run.smithi173.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [523 kB] 2023-11-01T04:01:39.906 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [523 kB] 2023-11-01T04:01:39.974 INFO:teuthology.orchestra.run.smithi173.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,148 kB] 2023-11-01T04:01:39.981 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,148 kB] 2023-11-01T04:01:39.986 INFO:teuthology.orchestra.run.smithi181.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [360 kB] 2023-11-01T04:01:39.991 INFO:teuthology.orchestra.run.smithi181.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [185 kB] 2023-11-01T04:01:39.993 INFO:teuthology.orchestra.run.smithi181.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-11-01T04:01:39.994 INFO:teuthology.orchestra.run.smithi181.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-11-01T04:01:39.994 INFO:teuthology.orchestra.run.smithi181.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,083 kB] 2023-11-01T04:01:40.002 INFO:teuthology.orchestra.run.smithi173.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [245 kB] 2023-11-01T04:01:40.006 INFO:teuthology.orchestra.run.smithi173.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-11-01T04:01:40.007 INFO:teuthology.orchestra.run.smithi173.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.4 kB] 2023-11-01T04:01:40.007 INFO:teuthology.orchestra.run.smithi173.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,103 kB] 2023-11-01T04:01:40.011 INFO:teuthology.orchestra.run.smithi181.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [176 kB] 2023-11-01T04:01:40.013 INFO:teuthology.orchestra.run.smithi181.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-11-01T04:01:40.014 INFO:teuthology.orchestra.run.smithi181.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [796 kB] 2023-11-01T04:01:40.014 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [245 kB] 2023-11-01T04:01:40.017 INFO:teuthology.orchestra.run.smithi173.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [179 kB] 2023-11-01T04:01:40.019 INFO:teuthology.orchestra.run.smithi173.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-11-01T04:01:40.019 INFO:teuthology.orchestra.run.smithi173.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [998 kB] 2023-11-01T04:01:40.020 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-11-01T04:01:40.020 INFO:teuthology.orchestra.run.smithi190.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,103 kB] 2023-11-01T04:01:40.022 INFO:teuthology.orchestra.run.smithi181.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [562 kB] 2023-11-01T04:01:40.025 INFO:teuthology.orchestra.run.smithi181.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [146 kB] 2023-11-01T04:01:40.026 INFO:teuthology.orchestra.run.smithi181.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-11-01T04:01:40.027 INFO:teuthology.orchestra.run.smithi173.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [660 kB] 2023-11-01T04:01:40.027 INFO:teuthology.orchestra.run.smithi181.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-11-01T04:01:40.031 INFO:teuthology.orchestra.run.smithi190.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.4 kB] 2023-11-01T04:01:40.032 INFO:teuthology.orchestra.run.smithi173.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [218 kB] 2023-11-01T04:01:40.032 INFO:teuthology.orchestra.run.smithi190.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [179 kB] 2023-11-01T04:01:40.033 INFO:teuthology.orchestra.run.smithi190.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-11-01T04:01:40.034 INFO:teuthology.orchestra.run.smithi190.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [998 kB] 2023-11-01T04:01:40.048 INFO:teuthology.orchestra.run.smithi173.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-11-01T04:01:40.048 INFO:teuthology.orchestra.run.smithi173.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-11-01T04:01:40.049 INFO:teuthology.orchestra.run.smithi181.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-11-01T04:01:40.049 INFO:teuthology.orchestra.run.smithi181.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-11-01T04:01:40.049 INFO:teuthology.orchestra.run.smithi181.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-11-01T04:01:40.050 INFO:teuthology.orchestra.run.smithi173.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-11-01T04:01:40.050 INFO:teuthology.orchestra.run.smithi173.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-11-01T04:01:40.050 INFO:teuthology.orchestra.run.smithi173.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-11-01T04:01:40.052 INFO:teuthology.orchestra.run.smithi173.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-11-01T04:01:40.053 INFO:teuthology.orchestra.run.smithi173.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-11-01T04:01:40.054 INFO:teuthology.orchestra.run.smithi173.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-11-01T04:01:40.054 INFO:teuthology.orchestra.run.smithi173.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-11-01T04:01:40.069 INFO:teuthology.orchestra.run.smithi173.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-11-01T04:01:40.070 INFO:teuthology.orchestra.run.smithi173.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-11-01T04:01:40.070 INFO:teuthology.orchestra.run.smithi173.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-11-01T04:01:40.070 INFO:teuthology.orchestra.run.smithi173.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-11-01T04:01:40.081 INFO:teuthology.orchestra.run.smithi190.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [660 kB] 2023-11-01T04:01:40.086 INFO:teuthology.orchestra.run.smithi190.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [218 kB] 2023-11-01T04:01:40.088 INFO:teuthology.orchestra.run.smithi190.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-11-01T04:01:40.089 INFO:teuthology.orchestra.run.smithi190.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-11-01T04:01:40.089 INFO:teuthology.orchestra.run.smithi190.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-11-01T04:01:40.090 INFO:teuthology.orchestra.run.smithi190.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-11-01T04:01:40.090 INFO:teuthology.orchestra.run.smithi190.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-11-01T04:01:40.091 INFO:teuthology.orchestra.run.smithi190.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-11-01T04:01:40.091 INFO:teuthology.orchestra.run.smithi190.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-11-01T04:01:40.102 INFO:teuthology.orchestra.run.smithi190.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-11-01T04:01:40.122 INFO:teuthology.orchestra.run.smithi190.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-11-01T04:01:40.123 INFO:teuthology.orchestra.run.smithi190.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-11-01T04:01:40.123 INFO:teuthology.orchestra.run.smithi190.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-11-01T04:01:40.124 INFO:teuthology.orchestra.run.smithi190.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-11-01T04:01:40.124 INFO:teuthology.orchestra.run.smithi190.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-11-01T04:01:40.201 INFO:teuthology.orchestra.run.smithi173.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [940 kB] 2023-11-01T04:01:40.203 INFO:teuthology.orchestra.run.smithi181.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-11-01T04:01:40.206 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-11-01T04:01:40.206 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [940 kB] 2023-11-01T04:01:40.231 INFO:teuthology.orchestra.run.smithi190.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [360 kB] 2023-11-01T04:01:40.342 INFO:teuthology.orchestra.run.smithi181.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [523 kB] 2023-11-01T04:01:40.349 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,148 kB] 2023-11-01T04:01:40.495 INFO:teuthology.orchestra.run.smithi190.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [940 kB] 2023-11-01T04:01:40.528 INFO:teuthology.orchestra.run.smithi181.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,148 kB] 2023-11-01T04:01:40.569 INFO:teuthology.orchestra.run.smithi173.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [360 kB] 2023-11-01T04:01:40.586 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [360 kB] 2023-11-01T04:01:40.590 INFO:teuthology.orchestra.run.smithi173.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [185 kB] 2023-11-01T04:01:40.600 INFO:teuthology.orchestra.run.smithi173.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-11-01T04:01:40.601 INFO:teuthology.orchestra.run.smithi173.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,083 kB] 2023-11-01T04:01:40.607 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [185 kB] 2023-11-01T04:01:40.619 INFO:teuthology.orchestra.run.smithi153.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-11-01T04:01:40.619 INFO:teuthology.orchestra.run.smithi153.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-11-01T04:01:40.621 INFO:teuthology.orchestra.run.smithi153.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,083 kB] 2023-11-01T04:01:40.622 INFO:teuthology.orchestra.run.smithi153.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [523 kB] 2023-11-01T04:01:40.644 INFO:teuthology.orchestra.run.smithi190.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [185 kB] 2023-11-01T04:01:40.656 INFO:teuthology.orchestra.run.smithi181.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [245 kB] 2023-11-01T04:01:40.660 INFO:teuthology.orchestra.run.smithi153.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [245 kB] 2023-11-01T04:01:40.663 INFO:teuthology.orchestra.run.smithi190.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-11-01T04:01:40.663 INFO:teuthology.orchestra.run.smithi190.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-11-01T04:01:40.665 INFO:teuthology.orchestra.run.smithi190.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,083 kB] 2023-11-01T04:01:40.673 INFO:teuthology.orchestra.run.smithi181.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-11-01T04:01:40.674 INFO:teuthology.orchestra.run.smithi181.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,103 kB] 2023-11-01T04:01:40.678 INFO:teuthology.orchestra.run.smithi153.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-11-01T04:01:40.679 INFO:teuthology.orchestra.run.smithi153.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,103 kB] 2023-11-01T04:01:40.687 INFO:teuthology.orchestra.run.smithi173.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-11-01T04:01:40.689 INFO:teuthology.orchestra.run.smithi173.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [176 kB] 2023-11-01T04:01:40.699 INFO:teuthology.orchestra.run.smithi173.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-11-01T04:01:40.700 INFO:teuthology.orchestra.run.smithi173.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [562 kB] 2023-11-01T04:01:40.706 INFO:teuthology.orchestra.run.smithi153.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [176 kB] 2023-11-01T04:01:40.716 INFO:teuthology.orchestra.run.smithi153.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-11-01T04:01:40.717 INFO:teuthology.orchestra.run.smithi153.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [796 kB] 2023-11-01T04:01:40.728 INFO:teuthology.orchestra.run.smithi181.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.4 kB] 2023-11-01T04:01:40.728 INFO:teuthology.orchestra.run.smithi173.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [796 kB] 2023-11-01T04:01:40.729 INFO:teuthology.orchestra.run.smithi181.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [179 kB] 2023-11-01T04:01:40.733 INFO:teuthology.orchestra.run.smithi153.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.4 kB] 2023-11-01T04:01:40.734 INFO:teuthology.orchestra.run.smithi153.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [179 kB] 2023-11-01T04:01:40.734 INFO:teuthology.orchestra.run.smithi190.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [176 kB] 2023-11-01T04:01:40.735 INFO:teuthology.orchestra.run.smithi181.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-11-01T04:01:40.740 INFO:teuthology.orchestra.run.smithi153.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-11-01T04:01:40.744 INFO:teuthology.orchestra.run.smithi190.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-11-01T04:01:40.745 INFO:teuthology.orchestra.run.smithi190.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [562 kB] 2023-11-01T04:01:40.754 INFO:teuthology.orchestra.run.smithi153.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [562 kB] 2023-11-01T04:01:40.766 INFO:teuthology.orchestra.run.smithi190.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [796 kB] 2023-11-01T04:01:40.901 INFO:teuthology.orchestra.run.smithi181.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [998 kB] 2023-11-01T04:01:40.904 INFO:teuthology.orchestra.run.smithi153.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [660 kB] 2023-11-01T04:01:40.915 INFO:teuthology.orchestra.run.smithi173.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [146 kB] 2023-11-01T04:01:40.933 INFO:teuthology.orchestra.run.smithi153.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [146 kB] 2023-11-01T04:01:40.955 INFO:teuthology.orchestra.run.smithi190.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [146 kB] 2023-11-01T04:01:41.299 INFO:teuthology.orchestra.run.smithi173.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-11-01T04:01:41.306 INFO:teuthology.orchestra.run.smithi173.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-11-01T04:01:41.306 INFO:teuthology.orchestra.run.smithi173.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-11-01T04:01:41.317 INFO:teuthology.orchestra.run.smithi153.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-11-01T04:01:41.321 INFO:teuthology.orchestra.run.smithi173.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-11-01T04:01:41.324 INFO:teuthology.orchestra.run.smithi153.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-11-01T04:01:41.325 INFO:teuthology.orchestra.run.smithi173.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-11-01T04:01:41.339 INFO:teuthology.orchestra.run.smithi153.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-11-01T04:01:41.340 INFO:teuthology.orchestra.run.smithi153.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-11-01T04:01:41.343 INFO:teuthology.orchestra.run.smithi190.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-11-01T04:01:41.343 INFO:teuthology.orchestra.run.smithi153.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-11-01T04:01:41.350 INFO:teuthology.orchestra.run.smithi190.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-11-01T04:01:41.365 INFO:teuthology.orchestra.run.smithi190.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-11-01T04:01:41.365 INFO:teuthology.orchestra.run.smithi190.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-11-01T04:01:41.368 INFO:teuthology.orchestra.run.smithi190.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-11-01T04:01:41.474 INFO:teuthology.orchestra.run.smithi153.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [998 kB] 2023-11-01T04:01:41.520 INFO:teuthology.orchestra.run.smithi181.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [660 kB] 2023-11-01T04:01:41.558 INFO:teuthology.orchestra.run.smithi181.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [218 kB] 2023-11-01T04:01:41.570 INFO:teuthology.orchestra.run.smithi181.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-11-01T04:01:41.572 INFO:teuthology.orchestra.run.smithi181.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-11-01T04:01:41.574 INFO:teuthology.orchestra.run.smithi181.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-11-01T04:01:41.574 INFO:teuthology.orchestra.run.smithi181.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-11-01T04:01:41.575 INFO:teuthology.orchestra.run.smithi181.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-11-01T04:01:41.575 INFO:teuthology.orchestra.run.smithi181.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-11-01T04:01:41.585 INFO:teuthology.orchestra.run.smithi153.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [218 kB] 2023-11-01T04:01:41.602 INFO:teuthology.orchestra.run.smithi181.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-11-01T04:01:41.603 INFO:teuthology.orchestra.run.smithi153.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-11-01T04:01:41.605 INFO:teuthology.orchestra.run.smithi181.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-11-01T04:01:41.605 INFO:teuthology.orchestra.run.smithi153.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-11-01T04:01:41.609 INFO:teuthology.orchestra.run.smithi153.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-11-01T04:01:41.609 INFO:teuthology.orchestra.run.smithi153.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-11-01T04:01:41.610 INFO:teuthology.orchestra.run.smithi153.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-11-01T04:01:41.610 INFO:teuthology.orchestra.run.smithi153.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-11-01T04:01:41.614 INFO:teuthology.orchestra.run.smithi153.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-11-01T04:01:41.619 INFO:teuthology.orchestra.run.smithi153.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-11-01T04:01:41.769 INFO:teuthology.orchestra.run.smithi181.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-11-01T04:01:41.783 INFO:teuthology.orchestra.run.smithi153.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-11-01T04:01:41.851 INFO:teuthology.orchestra.run.smithi181.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-11-01T04:01:41.866 INFO:teuthology.orchestra.run.smithi153.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-11-01T04:01:41.947 INFO:teuthology.orchestra.run.smithi153.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-11-01T04:01:41.953 INFO:teuthology.orchestra.run.smithi181.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-11-01T04:01:41.987 INFO:teuthology.orchestra.run.smithi181.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-11-01T04:01:42.001 INFO:teuthology.orchestra.run.smithi153.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-11-01T04:01:42.044 INFO:teuthology.orchestra.run.smithi181.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-11-01T04:01:42.059 INFO:teuthology.orchestra.run.smithi153.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-11-01T04:01:45.658 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 10.1 MB in 2s (5,617 kB/s) 2023-11-01T04:01:45.663 INFO:teuthology.orchestra.run.smithi173.stdout:Fetched 10.1 MB in 2s (5,776 kB/s) 2023-11-01T04:01:46.153 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 10.1 MB in 2s (4,057 kB/s) 2023-11-01T04:01:46.217 INFO:teuthology.orchestra.run.smithi181.stdout:Fetched 10.1 MB in 2s (4,102 kB/s) 2023-11-01T04:01:46.956 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2023-11-01T04:01:46.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-01T04:01:47.031 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2023-11-01T04:01:47.063 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2023-11-01T04:01:47.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-01T04:01:47.137 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2023-11-01T04:01:47.214 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2023-11-01T04:01:47.215 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2023-11-01T04:01:47.324 INFO:teuthology.orchestra.run.smithi173.stdout:Building dependency tree... 2023-11-01T04:01:47.325 INFO:teuthology.orchestra.run.smithi173.stdout:Reading state information... 2023-11-01T04:01:47.377 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages were automatically installed and are no longer required: 2023-11-01T04:01:47.378 INFO:teuthology.orchestra.run.smithi190.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-01T04:01:47.378 INFO:teuthology.orchestra.run.smithi190.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-01T04:01:47.379 INFO:teuthology.orchestra.run.smithi190.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-01T04:01:47.379 INFO:teuthology.orchestra.run.smithi190.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-11-01T04:01:47.379 INFO:teuthology.orchestra.run.smithi190.stdout: python2.7-minimal 2023-11-01T04:01:47.379 INFO:teuthology.orchestra.run.smithi190.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-01T04:01:47.380 INFO:teuthology.orchestra.run.smithi190.stdout:The following additional packages will be installed: 2023-11-01T04:01:47.380 INFO:teuthology.orchestra.run.smithi190.stdout: linux-generic linux-headers-5.15.0-88 linux-headers-5.15.0-88-generic 2023-11-01T04:01:47.380 INFO:teuthology.orchestra.run.smithi190.stdout: linux-headers-generic linux-image-5.15.0-88-generic 2023-11-01T04:01:47.381 INFO:teuthology.orchestra.run.smithi190.stdout: linux-modules-5.15.0-88-generic linux-modules-extra-5.15.0-88-generic 2023-11-01T04:01:47.381 INFO:teuthology.orchestra.run.smithi190.stdout:Suggested packages: 2023-11-01T04:01:47.381 INFO:teuthology.orchestra.run.smithi190.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-11-01T04:01:47.381 INFO:teuthology.orchestra.run.smithi190.stdout:Recommended packages: 2023-11-01T04:01:47.382 INFO:teuthology.orchestra.run.smithi190.stdout: thermald 2023-11-01T04:01:47.413 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2023-11-01T04:01:47.429 INFO:teuthology.orchestra.run.smithi190.stdout:The following NEW packages will be installed: 2023-11-01T04:01:47.429 INFO:teuthology.orchestra.run.smithi190.stdout: linux-headers-5.15.0-88 linux-headers-5.15.0-88-generic 2023-11-01T04:01:47.430 INFO:teuthology.orchestra.run.smithi190.stdout: linux-image-5.15.0-88-generic linux-modules-5.15.0-88-generic 2023-11-01T04:01:47.431 INFO:teuthology.orchestra.run.smithi190.stdout: linux-modules-extra-5.15.0-88-generic 2023-11-01T04:01:47.431 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-01T04:01:47.432 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages will be upgraded: 2023-11-01T04:01:47.432 INFO:teuthology.orchestra.run.smithi190.stdout: linux-generic linux-headers-generic linux-image-generic 2023-11-01T04:01:47.482 INFO:teuthology.orchestra.run.smithi173.stdout:The following packages were automatically installed and are no longer required: 2023-11-01T04:01:47.482 INFO:teuthology.orchestra.run.smithi173.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-01T04:01:47.482 INFO:teuthology.orchestra.run.smithi173.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-01T04:01:47.483 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2023-11-01T04:01:47.484 INFO:teuthology.orchestra.run.smithi173.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-01T04:01:47.484 INFO:teuthology.orchestra.run.smithi173.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-11-01T04:01:47.484 INFO:teuthology.orchestra.run.smithi173.stdout: python2.7-minimal 2023-11-01T04:01:47.484 INFO:teuthology.orchestra.run.smithi173.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-01T04:01:47.484 INFO:teuthology.orchestra.run.smithi173.stdout:The following additional packages will be installed: 2023-11-01T04:01:47.485 INFO:teuthology.orchestra.run.smithi173.stdout: linux-generic linux-headers-5.15.0-88 linux-headers-5.15.0-88-generic 2023-11-01T04:01:47.485 INFO:teuthology.orchestra.run.smithi173.stdout: linux-headers-generic linux-image-5.15.0-88-generic 2023-11-01T04:01:47.486 INFO:teuthology.orchestra.run.smithi190.stdout:3 upgraded, 5 newly installed, 0 to remove and 269 not upgraded. 2023-11-01T04:01:47.486 INFO:teuthology.orchestra.run.smithi190.stdout:Need to get 113 MB of archives. 2023-11-01T04:01:47.486 INFO:teuthology.orchestra.run.smithi190.stdout:After this operation, 582 MB of additional disk space will be used. 2023-11-01T04:01:47.486 INFO:teuthology.orchestra.run.smithi190.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-88-generic amd64 5.15.0-88.98 [22.4 MB] 2023-11-01T04:01:47.487 INFO:teuthology.orchestra.run.smithi173.stdout: linux-modules-5.15.0-88-generic linux-modules-extra-5.15.0-88-generic 2023-11-01T04:01:47.487 INFO:teuthology.orchestra.run.smithi173.stdout:Suggested packages: 2023-11-01T04:01:47.487 INFO:teuthology.orchestra.run.smithi173.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-11-01T04:01:47.487 INFO:teuthology.orchestra.run.smithi173.stdout:Recommended packages: 2023-11-01T04:01:47.487 INFO:teuthology.orchestra.run.smithi173.stdout: thermald 2023-11-01T04:01:47.488 INFO:teuthology.orchestra.run.smithi181.stdout:Reading package lists... 2023-11-01T04:01:47.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-01T04:01:47.530 INFO:teuthology.orchestra.run.smithi173.stdout:The following NEW packages will be installed: 2023-11-01T04:01:47.530 INFO:teuthology.orchestra.run.smithi173.stdout: linux-headers-5.15.0-88 linux-headers-5.15.0-88-generic 2023-11-01T04:01:47.530 INFO:teuthology.orchestra.run.smithi173.stdout: linux-image-5.15.0-88-generic linux-modules-5.15.0-88-generic 2023-11-01T04:01:47.531 INFO:teuthology.orchestra.run.smithi173.stdout: linux-modules-extra-5.15.0-88-generic 2023-11-01T04:01:47.532 INFO:teuthology.orchestra.run.smithi173.stdout:The following packages will be upgraded: 2023-11-01T04:01:47.532 INFO:teuthology.orchestra.run.smithi173.stdout: linux-generic linux-headers-generic linux-image-generic 2023-11-01T04:01:47.568 INFO:teuthology.orchestra.run.smithi181.stdout:Reading package lists... 2023-11-01T04:01:47.587 INFO:teuthology.orchestra.run.smithi173.stdout:3 upgraded, 5 newly installed, 0 to remove and 269 not upgraded. 2023-11-01T04:01:47.587 INFO:teuthology.orchestra.run.smithi173.stdout:Need to get 113 MB of archives. 2023-11-01T04:01:47.587 INFO:teuthology.orchestra.run.smithi173.stdout:After this operation, 582 MB of additional disk space will be used. 2023-11-01T04:01:47.587 INFO:teuthology.orchestra.run.smithi173.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-88-generic amd64 5.15.0-88.98 [22.4 MB] 2023-11-01T04:01:47.649 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2023-11-01T04:01:47.650 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2023-11-01T04:01:47.755 INFO:teuthology.orchestra.run.smithi181.stdout:Building dependency tree... 2023-11-01T04:01:47.755 INFO:teuthology.orchestra.run.smithi181.stdout:Reading state information... 2023-11-01T04:01:47.791 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages were automatically installed and are no longer required: 2023-11-01T04:01:47.792 INFO:teuthology.orchestra.run.smithi153.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-01T04:01:47.792 INFO:teuthology.orchestra.run.smithi153.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-01T04:01:47.793 INFO:teuthology.orchestra.run.smithi153.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-01T04:01:47.793 INFO:teuthology.orchestra.run.smithi153.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-11-01T04:01:47.793 INFO:teuthology.orchestra.run.smithi153.stdout: python2.7-minimal 2023-11-01T04:01:47.793 INFO:teuthology.orchestra.run.smithi153.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-01T04:01:47.794 INFO:teuthology.orchestra.run.smithi153.stdout:The following additional packages will be installed: 2023-11-01T04:01:47.794 INFO:teuthology.orchestra.run.smithi153.stdout: linux-generic linux-headers-5.15.0-88 linux-headers-5.15.0-88-generic 2023-11-01T04:01:47.794 INFO:teuthology.orchestra.run.smithi153.stdout: linux-headers-generic linux-image-5.15.0-88-generic 2023-11-01T04:01:47.795 INFO:teuthology.orchestra.run.smithi153.stdout: linux-modules-5.15.0-88-generic linux-modules-extra-5.15.0-88-generic 2023-11-01T04:01:47.795 INFO:teuthology.orchestra.run.smithi153.stdout:Suggested packages: 2023-11-01T04:01:47.795 INFO:teuthology.orchestra.run.smithi153.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-11-01T04:01:47.795 INFO:teuthology.orchestra.run.smithi153.stdout:Recommended packages: 2023-11-01T04:01:47.796 INFO:teuthology.orchestra.run.smithi153.stdout: thermald 2023-11-01T04:01:47.838 INFO:teuthology.orchestra.run.smithi153.stdout:The following NEW packages will be installed: 2023-11-01T04:01:47.839 INFO:teuthology.orchestra.run.smithi153.stdout: linux-headers-5.15.0-88 linux-headers-5.15.0-88-generic 2023-11-01T04:01:47.839 INFO:teuthology.orchestra.run.smithi153.stdout: linux-image-5.15.0-88-generic linux-modules-5.15.0-88-generic 2023-11-01T04:01:47.839 INFO:teuthology.orchestra.run.smithi153.stdout: linux-modules-extra-5.15.0-88-generic 2023-11-01T04:01:47.840 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages will be upgraded: 2023-11-01T04:01:47.840 INFO:teuthology.orchestra.run.smithi153.stdout: linux-generic linux-headers-generic linux-image-generic 2023-11-01T04:01:47.897 INFO:teuthology.orchestra.run.smithi153.stdout:3 upgraded, 5 newly installed, 0 to remove and 269 not upgraded. 2023-11-01T04:01:47.898 INFO:teuthology.orchestra.run.smithi153.stdout:Need to get 113 MB of archives. 2023-11-01T04:01:47.898 INFO:teuthology.orchestra.run.smithi153.stdout:After this operation, 582 MB of additional disk space will be used. 2023-11-01T04:01:47.898 INFO:teuthology.orchestra.run.smithi153.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-88-generic amd64 5.15.0-88.98 [22.4 MB] 2023-11-01T04:01:47.911 INFO:teuthology.orchestra.run.smithi190.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-88-generic amd64 5.15.0-88.98 [11.5 MB] 2023-11-01T04:01:47.916 INFO:teuthology.orchestra.run.smithi181.stdout:The following packages were automatically installed and are no longer required: 2023-11-01T04:01:47.917 INFO:teuthology.orchestra.run.smithi181.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-01T04:01:47.917 INFO:teuthology.orchestra.run.smithi181.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-01T04:01:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-01T04:01:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-11-01T04:01:47.918 INFO:teuthology.orchestra.run.smithi181.stdout: python2.7-minimal 2023-11-01T04:01:47.918 INFO:teuthology.orchestra.run.smithi181.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-01T04:01:47.919 INFO:teuthology.orchestra.run.smithi181.stdout:The following additional packages will be installed: 2023-11-01T04:01:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: linux-generic linux-headers-5.15.0-88 linux-headers-5.15.0-88-generic 2023-11-01T04:01:47.919 INFO:teuthology.orchestra.run.smithi181.stdout: linux-headers-generic linux-image-5.15.0-88-generic 2023-11-01T04:01:47.920 INFO:teuthology.orchestra.run.smithi181.stdout: linux-modules-5.15.0-88-generic linux-modules-extra-5.15.0-88-generic 2023-11-01T04:01:47.920 INFO:teuthology.orchestra.run.smithi181.stdout:Suggested packages: 2023-11-01T04:01:47.920 INFO:teuthology.orchestra.run.smithi181.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-11-01T04:01:47.920 INFO:teuthology.orchestra.run.smithi181.stdout:Recommended packages: 2023-11-01T04:01:47.921 INFO:teuthology.orchestra.run.smithi181.stdout: thermald 2023-11-01T04:01:47.961 INFO:teuthology.orchestra.run.smithi181.stdout:The following NEW packages will be installed: 2023-11-01T04:01:47.961 INFO:teuthology.orchestra.run.smithi181.stdout: linux-headers-5.15.0-88 linux-headers-5.15.0-88-generic 2023-11-01T04:01:47.962 INFO:teuthology.orchestra.run.smithi181.stdout: linux-image-5.15.0-88-generic linux-modules-5.15.0-88-generic 2023-11-01T04:01:47.962 INFO:teuthology.orchestra.run.smithi181.stdout: linux-modules-extra-5.15.0-88-generic 2023-11-01T04:01:47.963 INFO:teuthology.orchestra.run.smithi181.stdout:The following packages will be upgraded: 2023-11-01T04:01:47.963 INFO:teuthology.orchestra.run.smithi181.stdout: linux-generic linux-headers-generic linux-image-generic 2023-11-01T04:01:48.009 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-88-generic amd64 5.15.0-88.98 [64.0 MB] 2023-11-01T04:01:48.057 INFO:teuthology.orchestra.run.smithi173.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-88-generic amd64 5.15.0-88.98 [11.5 MB] 2023-11-01T04:01:48.142 INFO:teuthology.orchestra.run.smithi181.stdout:3 upgraded, 5 newly installed, 0 to remove and 269 not upgraded. 2023-11-01T04:01:48.143 INFO:teuthology.orchestra.run.smithi181.stdout:Need to get 113 MB of archives. 2023-11-01T04:01:48.143 INFO:teuthology.orchestra.run.smithi181.stdout:After this operation, 582 MB of additional disk space will be used. 2023-11-01T04:01:48.143 INFO:teuthology.orchestra.run.smithi181.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-88-generic amd64 5.15.0-88.98 [22.4 MB] 2023-11-01T04:01:48.270 INFO:teuthology.orchestra.run.smithi173.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-88-generic amd64 5.15.0-88.98 [64.0 MB] 2023-11-01T04:01:48.570 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-88-generic amd64 5.15.0-88.98 [11.5 MB] 2023-11-01T04:01:48.811 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.88.85 [1,698 B] 2023-11-01T04:01:48.811 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.88.85 [2,488 B] 2023-11-01T04:01:48.811 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-88 all 5.15.0-88.98 [12.3 MB] 2023-11-01T04:01:48.836 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-88-generic amd64 5.15.0-88.98 [64.0 MB] 2023-11-01T04:01:48.929 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-88-generic amd64 5.15.0-88.98 [2,878 kB] 2023-11-01T04:01:48.951 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.88.85 [2,340 B] 2023-11-01T04:01:49.343 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 113 MB in 2s (74.7 MB/s) 2023-11-01T04:01:49.615 INFO:teuthology.orchestra.run.smithi181.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-88-generic amd64 5.15.0-88.98 [11.5 MB] 2023-11-01T04:01:49.780 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-modules-5.15.0-88-generic. 2023-11-01T04:01:49.976 INFO:teuthology.orchestra.run.smithi181.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-88-generic amd64 5.15.0-88.98 [64.0 MB] 2023-11-01T04:01:50.175 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.88.85 [1,698 B] 2023-11-01T04:01:50.176 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.88.85 [2,488 B] 2023-11-01T04:01:50.176 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-88 all 5.15.0-88.98 [12.3 MB] 2023-11-01T04:01:50.179 INFO:teuthology.orchestra.run.smithi173.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.88.85 [1,698 B] 2023-11-01T04:01:50.179 INFO:teuthology.orchestra.run.smithi173.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.88.85 [2,488 B] 2023-11-01T04:01:50.180 INFO:teuthology.orchestra.run.smithi173.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-88 all 5.15.0-88.98 [12.3 MB] 2023-11-01T04:01:50.363 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-88-generic amd64 5.15.0-88.98 [2,878 kB] 2023-11-01T04:01:50.381 INFO:teuthology.orchestra.run.smithi173.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-88-generic amd64 5.15.0-88.98 [2,878 kB] 2023-11-01T04:01:50.411 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.88.85 [2,340 B] 2023-11-01T04:01:50.427 INFO:teuthology.orchestra.run.smithi173.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.88.85 [2,340 B] 2023-11-01T04:01:50.814 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 113 MB in 3s (44.1 MB/s) 2023-11-01T04:01:50.989 INFO:teuthology.orchestra.run.smithi173.stdout:Fetched 113 MB in 3s (39.2 MB/s) 2023-11-01T04:01:51.000 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-modules-5.15.0-88-generic. 2023-11-01T04:01:51.194 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package linux-modules-5.15.0-88-generic. 2023-11-01T04:01:51.807 INFO:teuthology.orchestra.run.smithi190.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-11-01T04:01:51.812 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../0-linux-modules-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-01T04:01:52.047 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-modules-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:01:52.184 INFO:teuthology.orchestra.run.smithi181.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.88.85 [1,698 B] 2023-11-01T04:01:52.185 INFO:teuthology.orchestra.run.smithi181.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.88.85 [2,488 B] 2023-11-01T04:01:52.185 INFO:teuthology.orchestra.run.smithi181.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-88 all 5.15.0-88.98 [12.3 MB] 2023-11-01T04:01:52.528 INFO:teuthology.orchestra.run.smithi181.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-88-generic amd64 5.15.0-88.98 [2,878 kB] 2023-11-01T04:01:52.611 INFO:teuthology.orchestra.run.smithi181.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.88.85 [2,340 B] 2023-11-01T04:01:52.973 INFO:teuthology.orchestra.run.smithi153.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-11-01T04:01:52.978 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../0-linux-modules-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-01T04:01:53.002 INFO:teuthology.orchestra.run.smithi181.stdout:Fetched 113 MB in 5s (24.4 MB/s) 2023-11-01T04:01:53.164 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-modules-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:01:53.218 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package linux-modules-5.15.0-88-generic. 2023-11-01T04:01:53.263 INFO:teuthology.orchestra.run.smithi173.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-11-01T04:01:53.267 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../0-linux-modules-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-01T04:01:53.375 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking linux-modules-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:01:54.155 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-image-5.15.0-88-generic. 2023-11-01T04:01:54.176 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../1-linux-image-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-01T04:01:54.265 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:01:54.758 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-88-generic. 2023-11-01T04:01:54.778 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-01T04:01:54.816 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-modules-extra-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:01:55.245 INFO:teuthology.orchestra.run.smithi181.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-11-01T04:01:55.250 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../0-linux-modules-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-01T04:01:55.341 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking linux-modules-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:01:55.440 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-image-5.15.0-88-generic. 2023-11-01T04:01:55.461 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../1-linux-image-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-01T04:01:55.539 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:01:55.617 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package linux-image-5.15.0-88-generic. 2023-11-01T04:01:55.638 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../1-linux-image-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-01T04:01:55.724 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:01:56.043 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-88-generic. 2023-11-01T04:01:56.064 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-01T04:01:56.101 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-modules-extra-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:01:56.237 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-88-generic. 2023-11-01T04:01:56.258 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-01T04:01:56.295 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking linux-modules-extra-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:01:57.850 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package linux-image-5.15.0-88-generic. 2023-11-01T04:01:57.871 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../1-linux-image-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-01T04:01:57.969 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:01:58.470 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-88-generic. 2023-11-01T04:01:58.490 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-01T04:01:58.528 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking linux-modules-extra-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:01.608 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../3-linux-generic_5.15.0.88.85_amd64.deb ... 2023-11-01T04:02:01.732 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-01T04:02:02.074 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.88.85_amd64.deb ... 2023-11-01T04:02:02.219 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-image-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-01T04:02:02.446 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../3-linux-generic_5.15.0.88.85_amd64.deb ... 2023-11-01T04:02:02.522 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-headers-5.15.0-88. 2023-11-01T04:02:02.544 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../5-linux-headers-5.15.0-88_5.15.0-88.98_all.deb ... 2023-11-01T04:02:02.570 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-01T04:02:02.588 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-headers-5.15.0-88 (5.15.0-88.98) ... 2023-11-01T04:02:02.896 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.88.85_amd64.deb ... 2023-11-01T04:02:03.017 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-image-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-01T04:02:03.303 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-headers-5.15.0-88. 2023-11-01T04:02:03.319 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../3-linux-generic_5.15.0.88.85_amd64.deb ... 2023-11-01T04:02:03.326 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../5-linux-headers-5.15.0-88_5.15.0-88.98_all.deb ... 2023-11-01T04:02:03.370 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-headers-5.15.0-88 (5.15.0-88.98) ... 2023-11-01T04:02:03.463 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking linux-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-01T04:02:03.820 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.88.85_amd64.deb ... 2023-11-01T04:02:03.941 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking linux-image-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-01T04:02:04.269 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package linux-headers-5.15.0-88. 2023-11-01T04:02:04.292 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../5-linux-headers-5.15.0-88_5.15.0-88.98_all.deb ... 2023-11-01T04:02:04.336 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking linux-headers-5.15.0-88 (5.15.0-88.98) ... 2023-11-01T04:02:05.056 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../3-linux-generic_5.15.0.88.85_amd64.deb ... 2023-11-01T04:02:05.163 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking linux-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-01T04:02:05.521 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.88.85_amd64.deb ... 2023-11-01T04:02:05.642 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking linux-image-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-01T04:02:05.970 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package linux-headers-5.15.0-88. 2023-11-01T04:02:05.992 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../5-linux-headers-5.15.0-88_5.15.0-88.98_all.deb ... 2023-11-01T04:02:06.036 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking linux-headers-5.15.0-88 (5.15.0-88.98) ... 2023-11-01T04:02:08.655 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-headers-5.15.0-88-generic. 2023-11-01T04:02:08.692 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../6-linux-headers-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-01T04:02:08.721 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-headers-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:08.787 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-headers-5.15.0-88-generic. 2023-11-01T04:02:08.823 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../6-linux-headers-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-01T04:02:08.862 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-headers-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:10.084 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package linux-headers-5.15.0-88-generic. 2023-11-01T04:02:10.122 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../6-linux-headers-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-01T04:02:10.167 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking linux-headers-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:11.332 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.88.85_amd64.deb ... 2023-11-01T04:02:11.458 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-headers-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-01T04:02:11.598 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.88.85_amd64.deb ... 2023-11-01T04:02:11.721 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-headers-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-01T04:02:11.816 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-headers-5.15.0-88 (5.15.0-88.98) ... 2023-11-01T04:02:11.938 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-headers-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:11.981 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package linux-headers-5.15.0-88-generic. 2023-11-01T04:02:12.019 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../6-linux-headers-5.15.0-88-generic_5.15.0-88.98_amd64.deb ... 2023-11-01T04:02:12.029 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-headers-5.15.0-88 (5.15.0-88.98) ... 2023-11-01T04:02:12.056 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking linux-headers-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:12.064 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-headers-generic (5.15.0.88.85) ... 2023-11-01T04:02:12.126 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-headers-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:12.191 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:12.260 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-headers-generic (5.15.0.88.85) ... 2023-11-01T04:02:12.387 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:13.483 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.88.85_amd64.deb ... 2023-11-01T04:02:13.515 INFO:teuthology.orchestra.run.smithi190.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-11-01T04:02:13.515 INFO:teuthology.orchestra.run.smithi190.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-11-01T04:02:13.516 INFO:teuthology.orchestra.run.smithi190.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-88-generic 2023-11-01T04:02:13.516 INFO:teuthology.orchestra.run.smithi190.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-88-generic 2023-11-01T04:02:13.594 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking linux-headers-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-01T04:02:13.679 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-modules-extra-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:13.683 INFO:teuthology.orchestra.run.smithi153.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-11-01T04:02:13.684 INFO:teuthology.orchestra.run.smithi153.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-11-01T04:02:13.684 INFO:teuthology.orchestra.run.smithi153.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-88-generic 2023-11-01T04:02:13.684 INFO:teuthology.orchestra.run.smithi153.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-88-generic 2023-11-01T04:02:13.843 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-modules-extra-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:13.910 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up linux-headers-5.15.0-88 (5.15.0-88.98) ... 2023-11-01T04:02:14.040 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up linux-headers-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:14.166 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up linux-headers-generic (5.15.0.88.85) ... 2023-11-01T04:02:14.310 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:14.734 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.88.85_amd64.deb ... 2023-11-01T04:02:14.859 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking linux-headers-generic (5.15.0.88.85) over (5.15.0.56.54) ... 2023-11-01T04:02:15.321 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-modules-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:15.568 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-modules-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:15.634 INFO:teuthology.orchestra.run.smithi173.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-11-01T04:02:15.635 INFO:teuthology.orchestra.run.smithi173.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-11-01T04:02:15.635 INFO:teuthology.orchestra.run.smithi173.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-88-generic 2023-11-01T04:02:15.635 INFO:teuthology.orchestra.run.smithi173.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-88-generic 2023-11-01T04:02:15.839 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up linux-modules-extra-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:15.850 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up linux-headers-5.15.0-88 (5.15.0-88.98) ... 2023-11-01T04:02:15.964 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up linux-headers-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:16.098 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up linux-headers-generic (5.15.0.88.85) ... 2023-11-01T04:02:16.226 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:16.970 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-image-generic (5.15.0.88.85) ... 2023-11-01T04:02:16.981 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-image-generic (5.15.0.88.85) ... 2023-11-01T04:02:17.095 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-generic (5.15.0.88.85) ... 2023-11-01T04:02:17.133 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-generic (5.15.0.88.85) ... 2023-11-01T04:02:17.222 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:17.259 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:17.264 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up linux-modules-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:17.290 INFO:teuthology.orchestra.run.smithi153.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-01T04:02:17.290 INFO:teuthology.orchestra.run.smithi153.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-88-generic 2023-11-01T04:02:17.330 INFO:teuthology.orchestra.run.smithi190.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-01T04:02:17.330 INFO:teuthology.orchestra.run.smithi190.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-88-generic 2023-11-01T04:02:17.540 INFO:teuthology.orchestra.run.smithi181.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-11-01T04:02:17.541 INFO:teuthology.orchestra.run.smithi181.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-11-01T04:02:17.541 INFO:teuthology.orchestra.run.smithi181.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-88-generic 2023-11-01T04:02:17.541 INFO:teuthology.orchestra.run.smithi181.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-88-generic 2023-11-01T04:02:17.730 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up linux-modules-extra-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:18.700 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up linux-image-generic (5.15.0.88.85) ... 2023-11-01T04:02:18.825 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up linux-generic (5.15.0.88.85) ... 2023-11-01T04:02:18.952 INFO:teuthology.orchestra.run.smithi173.stdout:Processing triggers for linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:19.029 INFO:teuthology.orchestra.run.smithi173.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-01T04:02:19.030 INFO:teuthology.orchestra.run.smithi173.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-88-generic 2023-11-01T04:02:19.130 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up linux-modules-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:20.557 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up linux-image-generic (5.15.0.88.85) ... 2023-11-01T04:02:20.683 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up linux-generic (5.15.0.88.85) ... 2023-11-01T04:02:20.826 INFO:teuthology.orchestra.run.smithi181.stdout:Processing triggers for linux-image-5.15.0-88-generic (5.15.0-88.98) ... 2023-11-01T04:02:21.607 INFO:teuthology.orchestra.run.smithi181.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-11-01T04:02:21.607 INFO:teuthology.orchestra.run.smithi181.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-88-generic 2023-11-01T04:02:31.853 INFO:teuthology.orchestra.run.smithi190.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-01T04:02:31.854 INFO:teuthology.orchestra.run.smithi190.stdout:Sourcing file `/etc/default/grub' 2023-11-01T04:02:31.872 INFO:teuthology.orchestra.run.smithi190.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-01T04:02:31.898 INFO:teuthology.orchestra.run.smithi153.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-01T04:02:31.898 INFO:teuthology.orchestra.run.smithi153.stdout:Sourcing file `/etc/default/grub' 2023-11-01T04:02:31.900 INFO:teuthology.orchestra.run.smithi190.stdout:Generating grub configuration file ... 2023-11-01T04:02:31.909 INFO:teuthology.orchestra.run.smithi153.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-01T04:02:31.934 INFO:teuthology.orchestra.run.smithi153.stdout:Generating grub configuration file ... 2023-11-01T04:02:32.170 INFO:teuthology.orchestra.run.smithi190.stdout:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-01T04:02:32.185 INFO:teuthology.orchestra.run.smithi190.stdout:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-01T04:02:32.193 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-01T04:02:32.212 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-01T04:02:32.457 INFO:teuthology.orchestra.run.smithi190.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-01T04:02:32.462 INFO:teuthology.orchestra.run.smithi190.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-01T04:02:32.510 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-01T04:02:32.516 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-01T04:02:32.533 INFO:teuthology.orchestra.run.smithi190.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-01T04:02:32.539 INFO:teuthology.orchestra.run.smithi190.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-01T04:02:32.595 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-01T04:02:32.600 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-01T04:02:32.714 INFO:teuthology.orchestra.run.smithi190.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-01T04:02:32.714 INFO:teuthology.orchestra.run.smithi190.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-11-01T04:02:32.714 INFO:teuthology.orchestra.run.smithi190.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-01T04:02:32.768 INFO:teuthology.orchestra.run.smithi153.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-01T04:02:32.769 INFO:teuthology.orchestra.run.smithi153.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-11-01T04:02:32.769 INFO:teuthology.orchestra.run.smithi153.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-01T04:02:32.772 INFO:teuthology.orchestra.run.smithi190.stdout:done 2023-11-01T04:02:32.812 INFO:teuthology.orchestra.run.smithi153.stdout:done 2023-11-01T04:02:33.229 INFO:teuthology.orchestra.run.smithi173.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-01T04:02:33.229 INFO:teuthology.orchestra.run.smithi173.stdout:Sourcing file `/etc/default/grub' 2023-11-01T04:02:33.244 DEBUG:teuthology.orchestra.run.smithi190:> dpkg -s linux-image-generic 2023-11-01T04:02:33.246 DEBUG:teuthology.orchestra.run.smithi153:> dpkg -s linux-image-generic 2023-11-01T04:02:33.259 INFO:teuthology.orchestra.run.smithi173.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-01T04:02:33.264 INFO:teuthology.orchestra.run.smithi153.stdout:Package: linux-image-generic 2023-11-01T04:02:33.265 INFO:teuthology.orchestra.run.smithi153.stdout:Status: install ok installed 2023-11-01T04:02:33.265 INFO:teuthology.orchestra.run.smithi153.stdout:Priority: optional 2023-11-01T04:02:33.265 INFO:teuthology.orchestra.run.smithi153.stdout:Section: kernel 2023-11-01T04:02:33.266 INFO:teuthology.orchestra.run.smithi153.stdout:Installed-Size: 20 2023-11-01T04:02:33.266 INFO:teuthology.orchestra.run.smithi153.stdout:Maintainer: Ubuntu Kernel Team 2023-11-01T04:02:33.266 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: amd64 2023-11-01T04:02:33.266 INFO:teuthology.orchestra.run.smithi153.stdout:Source: linux-meta 2023-11-01T04:02:33.266 INFO:teuthology.orchestra.run.smithi153.stdout:Version: 5.15.0.88.85 2023-11-01T04:02:33.267 INFO:teuthology.orchestra.run.smithi153.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-01T04:02:33.267 INFO:teuthology.orchestra.run.smithi153.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-01T04:02:33.267 INFO:teuthology.orchestra.run.smithi153.stdout:Recommends: thermald 2023-11-01T04:02:33.267 INFO:teuthology.orchestra.run.smithi153.stdout:Description: Generic Linux kernel image 2023-11-01T04:02:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: This package will always depend on the latest generic kernel image 2023-11-01T04:02:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: available. 2023-11-01T04:02:33.269 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-88-generic 2023-11-01T04:02:33.269 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-01T04:02:33.269 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-01T04:02:33.270 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-11-01T04:02:33.270 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2023-11-01T04:02:33.271 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get clean 2023-11-01T04:02:33.272 INFO:teuthology.orchestra.run.smithi190.stdout:Package: linux-image-generic 2023-11-01T04:02:33.273 INFO:teuthology.orchestra.run.smithi190.stdout:Status: install ok installed 2023-11-01T04:02:33.273 INFO:teuthology.orchestra.run.smithi190.stdout:Priority: optional 2023-11-01T04:02:33.273 INFO:teuthology.orchestra.run.smithi190.stdout:Section: kernel 2023-11-01T04:02:33.274 INFO:teuthology.orchestra.run.smithi190.stdout:Installed-Size: 20 2023-11-01T04:02:33.274 INFO:teuthology.orchestra.run.smithi190.stdout:Maintainer: Ubuntu Kernel Team 2023-11-01T04:02:33.274 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: amd64 2023-11-01T04:02:33.275 INFO:teuthology.orchestra.run.smithi190.stdout:Source: linux-meta 2023-11-01T04:02:33.275 INFO:teuthology.orchestra.run.smithi190.stdout:Version: 5.15.0.88.85 2023-11-01T04:02:33.275 INFO:teuthology.orchestra.run.smithi190.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-01T04:02:33.276 INFO:teuthology.orchestra.run.smithi190.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-01T04:02:33.276 INFO:teuthology.orchestra.run.smithi190.stdout:Recommends: thermald 2023-11-01T04:02:33.276 INFO:teuthology.orchestra.run.smithi190.stdout:Description: Generic Linux kernel image 2023-11-01T04:02:33.276 INFO:teuthology.orchestra.run.smithi190.stdout: This package will always depend on the latest generic kernel image 2023-11-01T04:02:33.277 INFO:teuthology.orchestra.run.smithi190.stdout: available. 2023-11-01T04:02:33.277 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-88-generic 2023-11-01T04:02:33.278 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-01T04:02:33.278 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-01T04:02:33.278 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2023-11-01T04:02:33.278 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2023-11-01T04:02:33.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get clean 2023-11-01T04:02:33.304 INFO:teuthology.orchestra.run.smithi173.stdout:Generating grub configuration file ... 2023-11-01T04:02:33.429 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2023-11-01T04:02:33.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get update 2023-11-01T04:02:33.551 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-01T04:02:33.558 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-01T04:02:33.575 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-01T04:02:33.600 INFO:teuthology.orchestra.run.smithi173.stdout:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-01T04:02:33.606 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-01T04:02:33.607 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-01T04:02:33.607 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-01T04:02:33.612 INFO:teuthology.orchestra.run.smithi173.stdout:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-01T04:02:33.623 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-01T04:02:33.667 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-01T04:02:33.953 INFO:teuthology.orchestra.run.smithi173.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-01T04:02:33.958 INFO:teuthology.orchestra.run.smithi173.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-01T04:02:34.031 INFO:teuthology.orchestra.run.smithi173.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-01T04:02:34.037 INFO:teuthology.orchestra.run.smithi173.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-01T04:02:34.168 INFO:teuthology.orchestra.run.smithi173.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-01T04:02:34.169 INFO:teuthology.orchestra.run.smithi173.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-11-01T04:02:34.169 INFO:teuthology.orchestra.run.smithi173.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-01T04:02:34.216 INFO:teuthology.orchestra.run.smithi173.stdout:done 2023-11-01T04:02:34.716 DEBUG:teuthology.orchestra.run.smithi173:> dpkg -s linux-image-generic 2023-11-01T04:02:34.786 INFO:teuthology.orchestra.run.smithi173.stdout:Package: linux-image-generic 2023-11-01T04:02:34.786 INFO:teuthology.orchestra.run.smithi173.stdout:Status: install ok installed 2023-11-01T04:02:34.787 INFO:teuthology.orchestra.run.smithi173.stdout:Priority: optional 2023-11-01T04:02:34.787 INFO:teuthology.orchestra.run.smithi173.stdout:Section: kernel 2023-11-01T04:02:34.787 INFO:teuthology.orchestra.run.smithi173.stdout:Installed-Size: 20 2023-11-01T04:02:34.790 INFO:teuthology.orchestra.run.smithi173.stdout:Maintainer: Ubuntu Kernel Team 2023-11-01T04:02:34.790 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: amd64 2023-11-01T04:02:34.791 INFO:teuthology.orchestra.run.smithi173.stdout:Source: linux-meta 2023-11-01T04:02:34.791 INFO:teuthology.orchestra.run.smithi173.stdout:Version: 5.15.0.88.85 2023-11-01T04:02:34.791 INFO:teuthology.orchestra.run.smithi173.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-01T04:02:34.791 INFO:teuthology.orchestra.run.smithi173.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-01T04:02:34.791 INFO:teuthology.orchestra.run.smithi173.stdout:Recommends: thermald 2023-11-01T04:02:34.792 INFO:teuthology.orchestra.run.smithi173.stdout:Description: Generic Linux kernel image 2023-11-01T04:02:34.792 INFO:teuthology.orchestra.run.smithi173.stdout: This package will always depend on the latest generic kernel image 2023-11-01T04:02:34.792 INFO:teuthology.orchestra.run.smithi173.stdout: available. 2023-11-01T04:02:34.792 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-88-generic 2023-11-01T04:02:34.793 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-01T04:02:34.793 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-01T04:02:34.793 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-11-01T04:02:34.793 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=None, version=distro) 2023-11-01T04:02:34.794 DEBUG:teuthology.orchestra.run.smithi173:> sudo apt-get clean 2023-11-01T04:02:34.834 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2023-11-01T04:02:34.851 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-01T04:02:34.910 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2023-11-01T04:02:34.927 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2023-11-01T04:02:34.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-01T04:02:34.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo apt-get update 2023-11-01T04:02:35.005 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2023-11-01T04:02:35.099 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2023-11-01T04:02:35.100 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2023-11-01T04:02:35.107 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-01T04:02:35.194 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2023-11-01T04:02:35.194 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2023-11-01T04:02:35.227 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-01T04:02:35.264 INFO:teuthology.orchestra.run.smithi153.stdout:linux-image-generic is already the newest version (5.15.0.88.85). 2023-11-01T04:02:35.264 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages were automatically installed and are no longer required: 2023-11-01T04:02:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-01T04:02:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-01T04:02:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-01T04:02:35.265 INFO:teuthology.orchestra.run.smithi153.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-01T04:02:35.266 INFO:teuthology.orchestra.run.smithi153.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-01T04:02:35.266 INFO:teuthology.orchestra.run.smithi153.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-01T04:02:35.305 INFO:teuthology.orchestra.run.smithi153.stdout:0 upgraded, 0 newly installed, 0 to remove and 269 not upgraded. 2023-11-01T04:02:35.307 DEBUG:teuthology.orchestra.run.smithi153:> dpkg -s linux-image-generic 2023-11-01T04:02:35.308 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-01T04:02:35.323 INFO:teuthology.orchestra.run.smithi153.stdout:Package: linux-image-generic 2023-11-01T04:02:35.323 INFO:teuthology.orchestra.run.smithi153.stdout:Status: install ok installed 2023-11-01T04:02:35.323 INFO:teuthology.orchestra.run.smithi153.stdout:Priority: optional 2023-11-01T04:02:35.323 INFO:teuthology.orchestra.run.smithi153.stdout:Section: kernel 2023-11-01T04:02:35.324 INFO:teuthology.orchestra.run.smithi153.stdout:Installed-Size: 20 2023-11-01T04:02:35.324 INFO:teuthology.orchestra.run.smithi153.stdout:Maintainer: Ubuntu Kernel Team 2023-11-01T04:02:35.324 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: amd64 2023-11-01T04:02:35.324 INFO:teuthology.orchestra.run.smithi153.stdout:Source: linux-meta 2023-11-01T04:02:35.324 INFO:teuthology.orchestra.run.smithi153.stdout:Version: 5.15.0.88.85 2023-11-01T04:02:35.325 INFO:teuthology.orchestra.run.smithi153.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-01T04:02:35.325 INFO:teuthology.orchestra.run.smithi153.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-01T04:02:35.325 INFO:teuthology.orchestra.run.smithi153.stdout:Recommends: thermald 2023-11-01T04:02:35.325 INFO:teuthology.orchestra.run.smithi153.stdout:Description: Generic Linux kernel image 2023-11-01T04:02:35.326 INFO:teuthology.orchestra.run.smithi153.stdout: This package will always depend on the latest generic kernel image 2023-11-01T04:02:35.326 INFO:teuthology.orchestra.run.smithi153.stdout: available. 2023-11-01T04:02:35.326 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2023-11-01T04:02:35.357 INFO:teuthology.orchestra.run.smithi190.stdout:linux-image-generic is already the newest version (5.15.0.88.85). 2023-11-01T04:02:35.357 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages were automatically installed and are no longer required: 2023-11-01T04:02:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-01T04:02:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-01T04:02:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-01T04:02:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-01T04:02:35.359 INFO:teuthology.orchestra.run.smithi190.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-01T04:02:35.359 INFO:teuthology.orchestra.run.smithi190.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-01T04:02:35.369 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.N9DJxKW0lT 2023-11-01T04:02:35.369 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub/grub.cfg /tmp/tmp.N9DJxKW0lT 2023-11-01T04:02:35.390 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-01T04:02:35.395 INFO:teuthology.orchestra.run.smithi190.stdout:0 upgraded, 0 newly installed, 0 to remove and 269 not upgraded. 2023-11-01T04:02:35.397 DEBUG:teuthology.orchestra.run.smithi190:> dpkg -s linux-image-generic 2023-11-01T04:02:35.412 INFO:teuthology.orchestra.run.smithi190.stdout:Package: linux-image-generic 2023-11-01T04:02:35.413 INFO:teuthology.orchestra.run.smithi190.stdout:Status: install ok installed 2023-11-01T04:02:35.413 INFO:teuthology.orchestra.run.smithi190.stdout:Priority: optional 2023-11-01T04:02:35.413 INFO:teuthology.orchestra.run.smithi190.stdout:Section: kernel 2023-11-01T04:02:35.413 INFO:teuthology.orchestra.run.smithi190.stdout:Installed-Size: 20 2023-11-01T04:02:35.413 INFO:teuthology.orchestra.run.smithi190.stdout:Maintainer: Ubuntu Kernel Team 2023-11-01T04:02:35.413 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: amd64 2023-11-01T04:02:35.414 INFO:teuthology.orchestra.run.smithi190.stdout:Source: linux-meta 2023-11-01T04:02:35.414 INFO:teuthology.orchestra.run.smithi190.stdout:Version: 5.15.0.88.85 2023-11-01T04:02:35.414 INFO:teuthology.orchestra.run.smithi190.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-01T04:02:35.414 INFO:teuthology.orchestra.run.smithi190.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-01T04:02:35.414 INFO:teuthology.orchestra.run.smithi190.stdout:Recommends: thermald 2023-11-01T04:02:35.415 INFO:teuthology.orchestra.run.smithi190.stdout:Description: Generic Linux kernel image 2023-11-01T04:02:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: This package will always depend on the latest generic kernel image 2023-11-01T04:02:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: available. 2023-11-01T04:02:35.415 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2023-11-01T04:02:35.429 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.N9DJxKW0lT 2023-11-01T04:02:35.457 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.xMv0iyKuc5 2023-11-01T04:02:35.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub/grub.cfg /tmp/tmp.xMv0iyKuc5 2023-11-01T04:02:35.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.xMv0iyKuc5 2023-11-01T04:02:35.537 INFO:teuthology.orchestra.run.smithi181.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-11-01T04:02:35.538 INFO:teuthology.orchestra.run.smithi181.stdout:Sourcing file `/etc/default/grub' 2023-11-01T04:02:35.552 INFO:teuthology.orchestra.run.smithi181.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-01T04:02:35.557 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.N9DJxKW0lT is 10KB 2023-11-01T04:02:35.571 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.N9DJxKW0lT 2023-11-01T04:02:35.575 INFO:teuthology.orchestra.run.smithi181.stdout:Generating grub configuration file ... 2023-11-01T04:02:35.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-01T04:02:35.632 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:02:35.632 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-01T04:02:35.633 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-01T04:02:35.635 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.xMv0iyKuc5 is 10KB 2023-11-01T04:02:35.647 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.xMv0iyKuc5 2023-11-01T04:02:35.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-01T04:02:35.704 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:02:35.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-01T04:02:35.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-01T04:02:35.712 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-88-generic 2023-11-01T04:02:35.713 DEBUG:teuthology.orchestra.run.smithi153:> sudo update-grub 2023-11-01T04:02:35.768 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-88-generic 2023-11-01T04:02:35.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo update-grub 2023-11-01T04:02:35.831 INFO:teuthology.orchestra.run.smithi181.stdout:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-01T04:02:35.850 INFO:teuthology.orchestra.run.smithi181.stdout:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-01T04:02:36.173 INFO:teuthology.orchestra.run.smithi181.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-01T04:02:36.178 INFO:teuthology.orchestra.run.smithi181.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-01T04:02:36.238 INFO:teuthology.orchestra.run.smithi181.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-01T04:02:36.244 INFO:teuthology.orchestra.run.smithi181.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-01T04:02:36.396 INFO:teuthology.orchestra.run.smithi181.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-01T04:02:36.396 INFO:teuthology.orchestra.run.smithi181.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-11-01T04:02:36.396 INFO:teuthology.orchestra.run.smithi181.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-01T04:02:36.432 INFO:teuthology.orchestra.run.smithi181.stdout:done 2023-11-01T04:02:36.454 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2023-11-01T04:02:36.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-01T04:02:36.531 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2023-11-01T04:02:36.718 INFO:teuthology.orchestra.run.smithi173.stdout:Building dependency tree... 2023-11-01T04:02:36.719 INFO:teuthology.orchestra.run.smithi173.stdout:Reading state information... 2023-11-01T04:02:36.826 INFO:teuthology.orchestra.run.smithi153.stderr:Sourcing file `/etc/default/grub' 2023-11-01T04:02:36.829 INFO:teuthology.orchestra.run.smithi153.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-01T04:02:36.830 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2023-11-01T04:02:36.881 INFO:teuthology.orchestra.run.smithi173.stdout:linux-image-generic is already the newest version (5.15.0.88.85). 2023-11-01T04:02:36.881 INFO:teuthology.orchestra.run.smithi173.stdout:The following packages were automatically installed and are no longer required: 2023-11-01T04:02:36.881 INFO:teuthology.orchestra.run.smithi173.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-01T04:02:36.882 INFO:teuthology.orchestra.run.smithi173.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-01T04:02:36.882 INFO:teuthology.orchestra.run.smithi173.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-01T04:02:36.882 INFO:teuthology.orchestra.run.smithi173.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-01T04:02:36.883 INFO:teuthology.orchestra.run.smithi173.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-01T04:02:36.883 INFO:teuthology.orchestra.run.smithi173.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-01T04:02:36.919 INFO:teuthology.orchestra.run.smithi173.stdout:0 upgraded, 0 newly installed, 0 to remove and 269 not upgraded. 2023-11-01T04:02:36.921 DEBUG:teuthology.orchestra.run.smithi173:> dpkg -s linux-image-generic 2023-11-01T04:02:36.933 DEBUG:teuthology.orchestra.run.smithi181:> dpkg -s linux-image-generic 2023-11-01T04:02:36.934 INFO:teuthology.orchestra.run.smithi173.stdout:Package: linux-image-generic 2023-11-01T04:02:36.934 INFO:teuthology.orchestra.run.smithi173.stdout:Status: install ok installed 2023-11-01T04:02:36.934 INFO:teuthology.orchestra.run.smithi173.stdout:Priority: optional 2023-11-01T04:02:36.934 INFO:teuthology.orchestra.run.smithi173.stdout:Section: kernel 2023-11-01T04:02:36.935 INFO:teuthology.orchestra.run.smithi173.stdout:Installed-Size: 20 2023-11-01T04:02:36.935 INFO:teuthology.orchestra.run.smithi173.stdout:Maintainer: Ubuntu Kernel Team 2023-11-01T04:02:36.935 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: amd64 2023-11-01T04:02:36.935 INFO:teuthology.orchestra.run.smithi173.stdout:Source: linux-meta 2023-11-01T04:02:36.935 INFO:teuthology.orchestra.run.smithi173.stdout:Version: 5.15.0.88.85 2023-11-01T04:02:36.935 INFO:teuthology.orchestra.run.smithi173.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-01T04:02:36.936 INFO:teuthology.orchestra.run.smithi173.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-01T04:02:36.936 INFO:teuthology.orchestra.run.smithi173.stdout:Recommends: thermald 2023-11-01T04:02:36.936 INFO:teuthology.orchestra.run.smithi173.stdout:Description: Generic Linux kernel image 2023-11-01T04:02:36.936 INFO:teuthology.orchestra.run.smithi173.stdout: This package will always depend on the latest generic kernel image 2023-11-01T04:02:36.936 INFO:teuthology.orchestra.run.smithi173.stdout: available. 2023-11-01T04:02:36.937 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2023-11-01T04:02:36.980 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.9utHnDomNi 2023-11-01T04:02:36.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub/grub.cfg /tmp/tmp.9utHnDomNi 2023-11-01T04:02:36.982 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-01T04:02:36.989 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-01T04:02:36.996 INFO:teuthology.orchestra.run.smithi190.stderr:Sourcing file `/etc/default/grub' 2023-11-01T04:02:36.998 INFO:teuthology.orchestra.run.smithi181.stdout:Package: linux-image-generic 2023-11-01T04:02:36.998 INFO:teuthology.orchestra.run.smithi181.stdout:Status: install ok installed 2023-11-01T04:02:36.998 INFO:teuthology.orchestra.run.smithi181.stdout:Priority: optional 2023-11-01T04:02:36.998 INFO:teuthology.orchestra.run.smithi181.stdout:Section: kernel 2023-11-01T04:02:36.999 INFO:teuthology.orchestra.run.smithi181.stdout:Installed-Size: 20 2023-11-01T04:02:36.999 INFO:teuthology.orchestra.run.smithi181.stdout:Maintainer: Ubuntu Kernel Team 2023-11-01T04:02:36.999 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: amd64 2023-11-01T04:02:36.999 INFO:teuthology.orchestra.run.smithi181.stdout:Source: linux-meta 2023-11-01T04:02:36.999 INFO:teuthology.orchestra.run.smithi181.stdout:Version: 5.15.0.88.85 2023-11-01T04:02:37.000 INFO:teuthology.orchestra.run.smithi181.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-01T04:02:37.000 INFO:teuthology.orchestra.run.smithi181.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-01T04:02:37.000 INFO:teuthology.orchestra.run.smithi181.stdout:Recommends: thermald 2023-11-01T04:02:37.000 INFO:teuthology.orchestra.run.smithi181.stdout:Description: Generic Linux kernel image 2023-11-01T04:02:37.000 INFO:teuthology.orchestra.run.smithi181.stdout: This package will always depend on the latest generic kernel image 2023-11-01T04:02:37.001 INFO:teuthology.orchestra.run.smithi181.stdout: available. 2023-11-01T04:02:37.001 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-88-generic 2023-11-01T04:02:37.001 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-01T04:02:37.001 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-01T04:02:37.001 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2023-11-01T04:02:37.002 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=None, version=distro) 2023-11-01T04:02:37.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo apt-get clean 2023-11-01T04:02:37.003 INFO:teuthology.orchestra.run.smithi190.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-01T04:02:37.003 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2023-11-01T04:02:37.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.9utHnDomNi 2023-11-01T04:02:37.142 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.9utHnDomNi is 10KB 2023-11-01T04:02:37.154 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.9utHnDomNi 2023-11-01T04:02:37.160 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-01T04:02:37.161 DEBUG:teuthology.orchestra.run.smithi181:> sudo apt-get update 2023-11-01T04:02:37.188 INFO:teuthology.orchestra.run.smithi190.stderr:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-01T04:02:37.195 INFO:teuthology.orchestra.run.smithi190.stderr:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-01T04:02:37.204 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-01T04:02:37.211 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-01T04:02:37.212 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:02:37.213 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-01T04:02:37.213 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-01T04:02:37.252 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-01T04:02:37.258 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-01T04:02:37.281 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-01T04:02:37.281 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-01T04:02:37.287 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-88-generic 2023-11-01T04:02:37.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo update-grub 2023-11-01T04:02:37.288 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-01T04:02:37.305 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-01T04:02:37.375 INFO:teuthology.orchestra.run.smithi153.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-01T04:02:37.375 INFO:teuthology.orchestra.run.smithi153.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-11-01T04:02:37.375 INFO:teuthology.orchestra.run.smithi153.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-01T04:02:37.397 INFO:teuthology.orchestra.run.smithi153.stderr:done 2023-11-01T04:02:37.405 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2023-11-01T04:02:37.465 INFO:teuthology.orchestra.run.smithi190.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-01T04:02:37.470 INFO:teuthology.orchestra.run.smithi190.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-01T04:02:37.509 INFO:teuthology.orchestra.run.smithi190.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-01T04:02:37.514 INFO:teuthology.orchestra.run.smithi190.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-01T04:02:37.638 INFO:teuthology.orchestra.run.smithi190.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-01T04:02:37.639 INFO:teuthology.orchestra.run.smithi190.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-11-01T04:02:37.639 INFO:teuthology.orchestra.run.smithi190.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-01T04:02:37.648 INFO:teuthology.orchestra.run.smithi190.stderr:done 2023-11-01T04:02:37.650 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2023-11-01T04:02:38.522 INFO:teuthology.orchestra.run.smithi173.stderr:Sourcing file `/etc/default/grub' 2023-11-01T04:02:38.523 INFO:teuthology.orchestra.run.smithi173.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-01T04:02:38.525 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2023-11-01T04:02:38.614 INFO:teuthology.orchestra.run.smithi181.stdout:Reading package lists... 2023-11-01T04:02:38.632 DEBUG:teuthology.orchestra.run.smithi181:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-11-01T04:02:38.683 INFO:teuthology.orchestra.run.smithi181.stdout:Reading package lists... 2023-11-01T04:02:38.761 INFO:teuthology.orchestra.run.smithi173.stderr:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-01T04:02:38.767 INFO:teuthology.orchestra.run.smithi173.stderr:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-01T04:02:38.856 INFO:teuthology.orchestra.run.smithi181.stdout:Building dependency tree... 2023-11-01T04:02:38.856 INFO:teuthology.orchestra.run.smithi181.stdout:Reading state information... 2023-11-01T04:02:38.999 INFO:teuthology.orchestra.run.smithi181.stdout:linux-image-generic is already the newest version (5.15.0.88.85). 2023-11-01T04:02:39.000 INFO:teuthology.orchestra.run.smithi181.stdout:The following packages were automatically installed and are no longer required: 2023-11-01T04:02:39.000 INFO:teuthology.orchestra.run.smithi181.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-01T04:02:39.000 INFO:teuthology.orchestra.run.smithi181.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-01T04:02:39.000 INFO:teuthology.orchestra.run.smithi181.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-01T04:02:39.000 INFO:teuthology.orchestra.run.smithi181.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-01T04:02:39.001 INFO:teuthology.orchestra.run.smithi181.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-01T04:02:39.001 INFO:teuthology.orchestra.run.smithi181.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-01T04:02:39.034 INFO:teuthology.orchestra.run.smithi181.stdout:0 upgraded, 0 newly installed, 0 to remove and 269 not upgraded. 2023-11-01T04:02:39.036 DEBUG:teuthology.orchestra.run.smithi181:> dpkg -s linux-image-generic 2023-11-01T04:02:39.040 INFO:teuthology.orchestra.run.smithi173.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-01T04:02:39.045 INFO:teuthology.orchestra.run.smithi173.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-01T04:02:39.089 INFO:teuthology.orchestra.run.smithi181.stdout:Package: linux-image-generic 2023-11-01T04:02:39.089 INFO:teuthology.orchestra.run.smithi181.stdout:Status: install ok installed 2023-11-01T04:02:39.089 INFO:teuthology.orchestra.run.smithi181.stdout:Priority: optional 2023-11-01T04:02:39.089 INFO:teuthology.orchestra.run.smithi181.stdout:Section: kernel 2023-11-01T04:02:39.090 INFO:teuthology.orchestra.run.smithi181.stdout:Installed-Size: 20 2023-11-01T04:02:39.090 INFO:teuthology.orchestra.run.smithi181.stdout:Maintainer: Ubuntu Kernel Team 2023-11-01T04:02:39.090 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: amd64 2023-11-01T04:02:39.090 INFO:teuthology.orchestra.run.smithi181.stdout:Source: linux-meta 2023-11-01T04:02:39.091 INFO:teuthology.orchestra.run.smithi181.stdout:Version: 5.15.0.88.85 2023-11-01T04:02:39.091 INFO:teuthology.orchestra.run.smithi181.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-88), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-11-01T04:02:39.091 INFO:teuthology.orchestra.run.smithi181.stdout:Depends: linux-image-5.15.0-88-generic, linux-modules-extra-5.15.0-88-generic, linux-firmware, intel-microcode, amd64-microcode 2023-11-01T04:02:39.091 INFO:teuthology.orchestra.run.smithi181.stdout:Recommends: thermald 2023-11-01T04:02:39.091 INFO:teuthology.orchestra.run.smithi181.stdout:Description: Generic Linux kernel image 2023-11-01T04:02:39.092 INFO:teuthology.orchestra.run.smithi181.stdout: This package will always depend on the latest generic kernel image 2023-11-01T04:02:39.092 INFO:teuthology.orchestra.run.smithi181.stdout: available. 2023-11-01T04:02:39.092 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2023-11-01T04:02:39.093 INFO:teuthology.orchestra.run.smithi173.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-01T04:02:39.094 INFO:teuthology.orchestra.run.smithi173.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-01T04:02:39.134 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.IyflauYgVK 2023-11-01T04:02:39.135 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub/grub.cfg /tmp/tmp.IyflauYgVK 2023-11-01T04:02:39.189 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.IyflauYgVK 2023-11-01T04:02:39.251 INFO:teuthology.orchestra.run.smithi173.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-01T04:02:39.251 INFO:teuthology.orchestra.run.smithi173.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-11-01T04:02:39.251 INFO:teuthology.orchestra.run.smithi173.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-01T04:02:39.269 INFO:teuthology.orchestra.run.smithi173.stderr:done 2023-11-01T04:02:39.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2023-11-01T04:02:39.316 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.IyflauYgVK is 10KB 2023-11-01T04:02:39.329 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.IyflauYgVK 2023-11-01T04:02:39.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-11-01T04:02:39.389 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:02:39.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-11-01T04:02:39.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-11-01T04:02:39.457 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-88-generic 2023-11-01T04:02:39.457 DEBUG:teuthology.orchestra.run.smithi181:> sudo update-grub 2023-11-01T04:02:40.724 INFO:teuthology.orchestra.run.smithi181.stderr:Sourcing file `/etc/default/grub' 2023-11-01T04:02:40.726 INFO:teuthology.orchestra.run.smithi181.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-11-01T04:02:40.728 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2023-11-01T04:02:40.930 INFO:teuthology.orchestra.run.smithi181.stderr:Found linux image: /boot/vmlinuz-5.15.0-88-generic 2023-11-01T04:02:40.936 INFO:teuthology.orchestra.run.smithi181.stderr:Found initrd image: /boot/initrd.img-5.15.0-88-generic 2023-11-01T04:02:41.174 INFO:teuthology.orchestra.run.smithi181.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-11-01T04:02:41.178 INFO:teuthology.orchestra.run.smithi181.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-11-01T04:02:41.219 INFO:teuthology.orchestra.run.smithi181.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-11-01T04:02:41.224 INFO:teuthology.orchestra.run.smithi181.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-11-01T04:02:41.365 INFO:teuthology.orchestra.run.smithi181.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-11-01T04:02:41.366 INFO:teuthology.orchestra.run.smithi181.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-11-01T04:02:41.366 INFO:teuthology.orchestra.run.smithi181.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-11-01T04:02:41.382 INFO:teuthology.orchestra.run.smithi181.stderr:done 2023-11-01T04:02:41.395 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2023-11-01T04:03:07.430 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-01T04:03:07.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-11-01T04:03:07.432 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:03:07.653 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2023-11-01T04:03:07.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-01T04:03:07.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:03:09.274 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-11-01T04:03:09.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2023-11-01T04:03:09.275 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:03:11.398 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2023-11-01T04:03:11.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2023-11-01T04:03:11.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:03:42.078 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2023-11-01T04:03:42.330 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2023-11-01T04:03:43.866 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2023-11-01T04:03:45.914 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2023-11-01T04:03:51.082 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-11-01T04:03:51.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:03:51.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-01T04:03:51.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:03:52.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2023-11-01T04:03:52.870 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:03:54.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2023-11-01T04:03:54.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:04:09.470 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2023-11-01T04:04:09.726 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2023-11-01T04:04:11.258 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2023-11-01T04:04:13.306 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2023-11-01T04:04:21.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-11-01T04:04:21.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:04:21.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-01T04:04:21.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:04:22.030 ERROR:teuthology.orchestra.connection:Error authenticating with smithi190.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 450, in connect self._auth( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 781, in _auth raise saved_exception File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 757, in _auth self._transport.auth_publickey(username, key) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1625, in auth_publickey raise SSHException("No existing session") paramiko.ssh_exception.SSHException: No existing session 2023-11-01T04:04:23.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2023-11-01T04:04:23.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:04:23.519 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2023-11-01T04:04:23.703 DEBUG:teuthology.orchestra.run.smithi173:> true 2023-11-01T04:04:24.717 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2023-11-01T04:04:24.718 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-88-generic"... 2023-11-01T04:04:24.718 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2023-11-01T04:04:24.764 INFO:teuthology.orchestra.run.smithi173.stdout:5.15.0-88-generic 2023-11-01T04:04:24.765 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-88-generic vs 5.15.0-88-generic 2023-11-01T04:04:24.765 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-01T04:04:24.765 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-01T04:04:25.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2023-11-01T04:04:25.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:04:25.523 DEBUG:teuthology.orchestra.run.smithi181:> true 2023-11-01T04:04:25.766 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-11-01T04:04:25.767 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-01T04:04:25.867 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2023-11-01T04:04:25.894 DEBUG:teuthology.parallel:result is None 2023-11-01T04:04:26.109 DEBUG:teuthology.orchestra.run.smithi190:> true 2023-11-01T04:04:26.129 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2023-11-01T04:04:26.129 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.15.0-88-generic"... 2023-11-01T04:04:26.130 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2023-11-01T04:04:26.174 INFO:teuthology.orchestra.run.smithi181.stdout:5.15.0-88-generic 2023-11-01T04:04:26.174 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-88-generic vs 5.15.0-88-generic 2023-11-01T04:04:26.174 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-01T04:04:26.175 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-01T04:04:27.099 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-01T04:04:27.099 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.15.0-88-generic"... 2023-11-01T04:04:27.100 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2023-11-01T04:04:27.145 INFO:teuthology.orchestra.run.smithi190.stdout:5.15.0-88-generic 2023-11-01T04:04:27.145 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-88-generic vs 5.15.0-88-generic 2023-11-01T04:04:27.145 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-01T04:04:27.146 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-01T04:04:27.176 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2023-11-01T04:04:27.176 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-01T04:04:27.236 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2023-11-01T04:04:27.267 DEBUG:teuthology.parallel:result is None 2023-11-01T04:04:28.146 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2023-11-01T04:04:28.147 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-01T04:04:28.223 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2023-11-01T04:04:28.239 DEBUG:teuthology.parallel:result is None 2023-11-01T04:04:38.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-11-01T04:04:38.521 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:04:38.715 DEBUG:teuthology.orchestra.run.smithi153:> true 2023-11-01T04:04:39.358 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-11-01T04:04:39.358 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-88-generic"... 2023-11-01T04:04:39.359 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2023-11-01T04:04:39.406 INFO:teuthology.orchestra.run.smithi153.stdout:5.15.0-88-generic 2023-11-01T04:04:39.406 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-88-generic vs 5.15.0-88-generic 2023-11-01T04:04:39.407 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-01T04:04:39.407 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-11-01T04:04:40.407 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-01T04:04:40.408 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-01T04:04:40.731 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2023-11-01T04:04:40.755 DEBUG:teuthology.parallel:result is None 2023-11-01T04:04:40.756 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-01T04:04:40.762 INFO:teuthology.task.internal:Creating test directory... 2023-11-01T04:04:40.763 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-01T04:04:40.765 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-01T04:04:40.768 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-01T04:04:40.771 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-01T04:04:40.777 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-01T04:04:40.783 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-01T04:04:40.793 INFO:teuthology.task.internal:Creating archive directory... 2023-11-01T04:04:40.793 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-01T04:04:40.815 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-01T04:04:40.818 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-01T04:04:40.820 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-01T04:04:40.846 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-01T04:04:40.852 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-01T04:04:40.853 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-01T04:04:40.871 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-01T04:04:40.877 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-01T04:04:40.880 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-01T04:04:40.892 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-01T04:04:40.898 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-01T04:04:40.898 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-01T04:04:40.901 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-01T04:04:40.904 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-01T04:04:40.905 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-01T04:04:40.906 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-01T04:04:40.911 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-01T04:04:40.913 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-01T04:04:40.920 INFO:teuthology.task.internal:Configuring sudo... 2023-11-01T04:04:40.920 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-01T04:04:40.955 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-01T04:04:40.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-01T04:04:40.960 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-01T04:04:40.984 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-01T04:04:40.993 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-01T04:04:40.994 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-01T04:04:41.011 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-01T04:04:41.014 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-01T04:04:41.027 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-01T04:04:41.033 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-01T04:04:41.062 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-01T04:04:41.110 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:04:41.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-01T04:04:41.195 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-01T04:04:41.201 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-01T04:04:41.248 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:04:41.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-01T04:04:41.310 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-01T04:04:41.316 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-01T04:04:41.362 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:04:41.363 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-01T04:04:41.434 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-01T04:04:41.440 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-01T04:04:41.485 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:04:41.486 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-01T04:04:41.548 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2023-11-01T04:04:41.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2023-11-01T04:04:41.553 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2023-11-01T04:04:41.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2023-11-01T04:04:41.640 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-01T04:04:41.648 INFO:teuthology.task.internal:Starting timer... 2023-11-01T04:04:41.648 INFO:teuthology.run_tasks:Running task pcp... 2023-11-01T04:04:41.658 INFO:teuthology.run_tasks:Running task selinux... 2023-11-01T04:04:41.666 DEBUG:teuthology.task.selinux:Excluding smithi153: OS 'ubuntu' does not support SELinux 2023-11-01T04:04:41.667 DEBUG:teuthology.task.selinux:Excluding smithi173: OS 'ubuntu' does not support SELinux 2023-11-01T04:04:41.667 DEBUG:teuthology.task.selinux:Excluding smithi181: OS 'ubuntu' does not support SELinux 2023-11-01T04:04:41.667 DEBUG:teuthology.task.selinux:Excluding smithi190: OS 'ubuntu' does not support SELinux 2023-11-01T04:04:41.667 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-01T04:04:41.668 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-11-01T04:04:41.668 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-01T04:04:41.668 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-01T04:04:41.677 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-11-01T04:04:41.682 INFO:teuthology.repo_utils:Fetching main from origin 2023-11-01T04:04:41.804 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-01T04:04:41.819 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-01T04:04:41.821 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi153.front.sepia.ceph.com,smithi173.front.sepia.ceph.com,smithi181.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-01T04:12:20.760 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2023-11-01T04:12:20.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-11-01T04:12:20.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:12:20.842 DEBUG:teuthology.orchestra.run.smithi153:> true 2023-11-01T04:12:20.942 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-11-01T04:12:20.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2023-11-01T04:12:20.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:12:21.017 DEBUG:teuthology.orchestra.run.smithi173:> true 2023-11-01T04:12:21.130 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2023-11-01T04:12:21.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2023-11-01T04:12:21.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:12:21.210 DEBUG:teuthology.orchestra.run.smithi181:> true 2023-11-01T04:12:21.314 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2023-11-01T04:12:21.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-01T04:12:21.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T04:12:21.388 DEBUG:teuthology.orchestra.run.smithi190:> true 2023-11-01T04:12:21.498 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-11-01T04:12:21.500 INFO:teuthology.run_tasks:Running task clock... 2023-11-01T04:12:21.510 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-01T04:12:21.511 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-01T04:12:21.511 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-01T04:12:21.514 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-01T04:12:21.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-01T04:12:21.517 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-01T04:12:21.517 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-01T04:12:21.520 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-01T04:12:21.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-01T04:12:21.540 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-11-01T04:12:21.541 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: Command line: ntpd -gq 2023-11-01T04:12:21.541 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: ---------------------------------------------------- 2023-11-01T04:12:21.541 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: ntp-4 is maintained by Network Time Foundation, 2023-11-01T04:12:21.541 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-11-01T04:12:21.541 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: corporation. Support and training for ntp-4 are 2023-11-01T04:12:21.542 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: available at https://www.nwtime.org/support 2023-11-01T04:12:21.542 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: ---------------------------------------------------- 2023-11-01T04:12:21.543 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: proto: precision = 0.044 usec (-24) 2023-11-01T04:12:21.543 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: basedate set to 2022-02-04 2023-11-01T04:12:21.544 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: gps base set to 2022-02-06 (week 2196) 2023-11-01T04:12:21.544 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-01T04:12:21.544 INFO:teuthology.orchestra.run.smithi153.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-01T04:12:21.544 INFO:teuthology.orchestra.run.smithi153.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-01T04:12:21.545 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: restrict ::: KOD does nothing without LIMITED. 2023-11-01T04:12:21.546 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: Listen and drop on 0 v6wildcard [::]:123 2023-11-01T04:12:21.546 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-01T04:12:21.546 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: Listen normally on 2 lo 127.0.0.1:123 2023-11-01T04:12:21.546 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: Listen normally on 3 enp3s0f1 172.21.15.153:123 2023-11-01T04:12:21.546 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: Listen normally on 4 lo [::1]:123 2023-11-01T04:12:21.547 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6deb%5]:123 2023-11-01T04:12:21.547 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:21 ntpd[28003]: Listening on routing socket on fd #22 for interface updates 2023-11-01T04:12:21.548 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-11-01T04:12:21.548 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: Command line: ntpd -gq 2023-11-01T04:12:21.548 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: ---------------------------------------------------- 2023-11-01T04:12:21.548 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: ntp-4 is maintained by Network Time Foundation, 2023-11-01T04:12:21.549 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-11-01T04:12:21.549 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: corporation. Support and training for ntp-4 are 2023-11-01T04:12:21.549 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: available at https://www.nwtime.org/support 2023-11-01T04:12:21.549 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: ---------------------------------------------------- 2023-11-01T04:12:21.549 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: proto: precision = 0.060 usec (-24) 2023-11-01T04:12:21.550 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: basedate set to 2022-02-04 2023-11-01T04:12:21.550 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: gps base set to 2022-02-06 (week 2196) 2023-11-01T04:12:21.550 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-01T04:12:21.550 INFO:teuthology.orchestra.run.smithi173.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-01T04:12:21.551 INFO:teuthology.orchestra.run.smithi173.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-01T04:12:21.551 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: restrict ::: KOD does nothing without LIMITED. 2023-11-01T04:12:21.552 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: Listen and drop on 0 v6wildcard [::]:123 2023-11-01T04:12:21.552 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-01T04:12:21.552 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: Listen normally on 2 lo 127.0.0.1:123 2023-11-01T04:12:21.552 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: Listen normally on 3 enp3s0f1 172.21.15.173:123 2023-11-01T04:12:21.552 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: Listen normally on 4 lo [::1]:123 2023-11-01T04:12:21.553 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7259%5]:123 2023-11-01T04:12:21.553 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:21 ntpd[28005]: Listening on routing socket on fd #22 for interface updates 2023-11-01T04:12:21.554 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-11-01T04:12:21.554 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: Command line: ntpd -gq 2023-11-01T04:12:21.554 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: ---------------------------------------------------- 2023-11-01T04:12:21.554 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: ntp-4 is maintained by Network Time Foundation, 2023-11-01T04:12:21.554 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-11-01T04:12:21.555 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: corporation. Support and training for ntp-4 are 2023-11-01T04:12:21.555 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: available at https://www.nwtime.org/support 2023-11-01T04:12:21.555 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: ---------------------------------------------------- 2023-11-01T04:12:21.555 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: proto: precision = 0.044 usec (-24) 2023-11-01T04:12:21.555 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: basedate set to 2022-02-04 2023-11-01T04:12:21.556 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: gps base set to 2022-02-06 (week 2196) 2023-11-01T04:12:21.556 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-01T04:12:21.556 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: restrict ::: KOD does nothing without LIMITED. 2023-11-01T04:12:21.556 INFO:teuthology.orchestra.run.smithi181.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-01T04:12:21.556 INFO:teuthology.orchestra.run.smithi181.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-01T04:12:21.557 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: Listen and drop on 0 v6wildcard [::]:123 2023-11-01T04:12:21.558 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-01T04:12:21.558 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: Listen normally on 2 lo 127.0.0.1:123 2023-11-01T04:12:21.558 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: Listen normally on 3 enp3s0f1 172.21.15.181:123 2023-11-01T04:12:21.558 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: Listen normally on 4 lo [::1]:123 2023-11-01T04:12:21.558 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7b9d%5]:123 2023-11-01T04:12:21.559 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:21 ntpd[27976]: Listening on routing socket on fd #22 for interface updates 2023-11-01T04:12:21.564 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-11-01T04:12:21.564 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: Command line: ntpd -gq 2023-11-01T04:12:21.565 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: ---------------------------------------------------- 2023-11-01T04:12:21.565 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: ntp-4 is maintained by Network Time Foundation, 2023-11-01T04:12:21.565 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-11-01T04:12:21.565 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: corporation. Support and training for ntp-4 are 2023-11-01T04:12:21.565 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: available at https://www.nwtime.org/support 2023-11-01T04:12:21.566 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: ---------------------------------------------------- 2023-11-01T04:12:21.566 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: proto: precision = 0.045 usec (-24) 2023-11-01T04:12:21.567 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: basedate set to 2022-02-04 2023-11-01T04:12:21.567 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: gps base set to 2022-02-06 (week 2196) 2023-11-01T04:12:21.567 INFO:teuthology.orchestra.run.smithi190.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-01T04:12:21.568 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-11-01T04:12:21.568 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: restrict ::: KOD does nothing without LIMITED. 2023-11-01T04:12:21.568 INFO:teuthology.orchestra.run.smithi190.stderr:restrict ::: KOD does nothing without LIMITED. 2023-11-01T04:12:21.569 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: Listen and drop on 0 v6wildcard [::]:123 2023-11-01T04:12:21.569 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-11-01T04:12:21.570 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: Listen normally on 2 lo 127.0.0.1:123 2023-11-01T04:12:21.570 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: Listen normally on 3 enp3s0f1 172.21.15.190:123 2023-11-01T04:12:21.570 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: Listen normally on 4 lo [::1]:123 2023-11-01T04:12:21.570 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7763%5]:123 2023-11-01T04:12:21.570 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:21 ntpd[28048]: Listening on routing socket on fd #22 for interface updates 2023-11-01T04:12:22.541 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:22 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:22.542 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:22 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:22.542 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:22 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:22.542 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:22 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:22.542 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:22 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:22.542 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:22 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:22.543 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:22 ntpd[28003]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:22.543 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:22 ntpd[28003]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:22.545 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:22 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:22.545 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:22 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:22.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:22 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:22.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:22 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:22.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:22 ntpd[28005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:22.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:22 ntpd[28005]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:22.565 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:22 ntpd[28048]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:22.565 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:22 ntpd[28048]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:22.566 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:22 ntpd[28048]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:22.566 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:22 ntpd[28048]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:22.566 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:22 ntpd[28048]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:22.566 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:22 ntpd[28048]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:23.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:23 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:23.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:23 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:23.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:23 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:23.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:23 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:23.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:23 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:23.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:23 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:23.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:23 ntpd[28005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:23.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:23 ntpd[28005]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:23.549 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:23 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:23.550 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:23 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:23.550 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:23 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:23.550 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:23 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:23.550 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:23 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:23.550 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:23 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:23.565 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:23 ntpd[28048]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:23.565 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:23 ntpd[28048]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:24.542 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:24 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:24.542 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:24 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:24.542 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:24 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:24.542 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:24 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:24.543 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:24 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:24.543 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:24 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:24.543 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:24 ntpd[28003]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:24.543 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:24 ntpd[28003]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:24.543 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:24 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:24.543 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:24 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:24.544 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:24 ntpd[28003]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:24.544 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:24 ntpd[28003]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:24.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:24 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:24.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:24 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:24.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:24 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:24.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:24 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:24.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:24 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:24.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:24 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:24.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:24 ntpd[28005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:24.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:24 ntpd[28005]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:24.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:24 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:24.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:24 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:24.548 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:24 ntpd[28005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:24.548 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:24 ntpd[28005]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:24.565 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:24 ntpd[28048]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:24.565 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:24 ntpd[28048]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:24.565 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:24 ntpd[28048]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:24.566 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:24 ntpd[28048]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:24.566 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:24 ntpd[28048]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:24.566 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:24 ntpd[28048]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:25.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:25 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:25.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:25 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:25.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:25 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:25.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:25 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:25.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:25 ntpd[28005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:25.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:25 ntpd[28005]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:25.550 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:25.551 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:25.551 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:25.551 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:25.551 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:25.551 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:25.552 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:25.552 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:25.552 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:25.552 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:25.552 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:25.553 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:25.553 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:25.553 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:25.553 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:25.553 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:25.553 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:25.554 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:25 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:26.542 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.543 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.543 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.543 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.543 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.543 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.544 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.544 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.544 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:26.544 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:26.544 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.545 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.545 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:26.545 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:26.545 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.545 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.545 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:26.546 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:26 ntpd[28003]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:26.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:26 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:26 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:26 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:26 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.548 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:26 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.548 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:26 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.548 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:26 ntpd[28005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:26.548 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:26 ntpd[28005]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:26.548 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:26 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.548 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:26 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.549 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:26 ntpd[28005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:26.549 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:26 ntpd[28005]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:26.549 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:26 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.550 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:26 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.566 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.566 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.566 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.566 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.567 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.567 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.567 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.567 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.567 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:26.567 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:26.568 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.568 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.568 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:26.568 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:26.568 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:26.568 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:26.569 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:26.569 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:26 ntpd[28048]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:27.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:27 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:27.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:27 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:27.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:27 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:27.546 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:27 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:27.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:27 ntpd[28005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:27.547 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:27 ntpd[28005]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:27.550 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:27.550 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:27.550 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:27.550 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:27.550 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:27.551 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:27.551 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:27.551 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:27.551 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:27.551 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:27.552 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:27.552 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:27.552 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:27.552 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:27.552 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:27.552 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:27.553 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:27.553 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:27 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:28.543 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:28 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:28.544 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:28 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:28.544 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:28 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:28.544 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:28 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:28.544 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:28 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:28.544 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:28 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:28.545 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:28 ntpd[28003]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:28.545 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:28 ntpd[28003]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:28.545 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:28 ntpd[28003]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:28.545 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:28 ntpd[28003]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:28.546 INFO:teuthology.orchestra.run.smithi153.stdout: 1 Nov 04:12:28 ntpd[28003]: ntpd: time slew +0.000062 s 2023-11-01T04:12:28.546 INFO:teuthology.orchestra.run.smithi153.stdout:ntpd: time slew +0.000062s 2023-11-01T04:12:28.546 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:28 ntpd[28003]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-01T04:12:28.547 INFO:teuthology.orchestra.run.smithi153.stderr: 1 Nov 04:12:28 ntpd[28003]: can't open /var/log/ntpstats/loopstats.20231101: Permission denied 2023-11-01T04:12:28.548 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:28 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:28.548 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:28 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:28.548 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:28 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:28.549 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:28 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:28.549 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:28 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:28.549 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:28 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:28.549 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:28 ntpd[28005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:28.549 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:28 ntpd[28005]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:28.550 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:28 ntpd[28005]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:28.550 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:28 ntpd[28005]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:28.550 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:28 ntpd[28005]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:28.550 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:28 ntpd[28005]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:28.551 INFO:teuthology.orchestra.run.smithi173.stdout: 1 Nov 04:12:28 ntpd[28005]: ntpd: time slew -0.000762 s 2023-11-01T04:12:28.551 INFO:teuthology.orchestra.run.smithi173.stdout:ntpd: time slew -0.000762s 2023-11-01T04:12:28.551 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:28 ntpd[28005]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-01T04:12:28.551 INFO:teuthology.orchestra.run.smithi173.stderr: 1 Nov 04:12:28 ntpd[28005]: can't open /var/log/ntpstats/loopstats.20231101: Permission denied 2023-11-01T04:12:28.565 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:28 ntpd[28048]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:28.566 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:28 ntpd[28048]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:28.566 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:28 ntpd[28048]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:28.566 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:28 ntpd[28048]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:28.566 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:28 ntpd[28048]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:28.566 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:28 ntpd[28048]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:28.567 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:28 ntpd[28048]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:28.567 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:28 ntpd[28048]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:28.567 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:28 ntpd[28048]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:28.567 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:28 ntpd[28048]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:28.568 INFO:teuthology.orchestra.run.smithi190.stdout: 1 Nov 04:12:28 ntpd[28048]: ntpd: time slew -0.000763 s 2023-11-01T04:12:28.568 INFO:teuthology.orchestra.run.smithi190.stdout:ntpd: time slew -0.000763s 2023-11-01T04:12:28.568 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:28 ntpd[28048]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-01T04:12:28.568 INFO:teuthology.orchestra.run.smithi190.stderr: 1 Nov 04:12:28 ntpd[28048]: can't open /var/log/ntpstats/loopstats.20231101: Permission denied 2023-11-01T04:12:28.631 INFO:teuthology.orchestra.run.smithi153.stdout: remote refid st t when poll reach delay offset jitter 2023-11-01T04:12:28.631 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================== 2023-11-01T04:12:28.631 INFO:teuthology.orchestra.run.smithi153.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-01T04:12:28.631 INFO:teuthology.orchestra.run.smithi153.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-01T04:12:28.631 INFO:teuthology.orchestra.run.smithi153.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-01T04:12:28.632 INFO:teuthology.orchestra.run.smithi153.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-01T04:12:28.638 INFO:teuthology.orchestra.run.smithi173.stdout: remote refid st t when poll reach delay offset jitter 2023-11-01T04:12:28.638 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================== 2023-11-01T04:12:28.638 INFO:teuthology.orchestra.run.smithi173.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-01T04:12:28.638 INFO:teuthology.orchestra.run.smithi173.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-01T04:12:28.639 INFO:teuthology.orchestra.run.smithi173.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-01T04:12:28.639 INFO:teuthology.orchestra.run.smithi173.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-01T04:12:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: remote refid st t when poll reach delay offset jitter 2023-11-01T04:12:28.647 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================== 2023-11-01T04:12:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-01T04:12:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-01T04:12:28.647 INFO:teuthology.orchestra.run.smithi190.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-01T04:12:28.648 INFO:teuthology.orchestra.run.smithi190.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-01T04:12:29.550 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:29.550 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:29.550 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:29.551 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:29.551 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:29.551 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:29.551 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:29.551 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:29.552 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:29.552 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:29.552 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:29.552 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:29.552 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:29.553 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:29.553 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-11-01T04:12:29.553 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: can't open /var/log/ntpstats/rawstats.20231101: Permission denied 2023-11-01T04:12:29.553 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-11-01T04:12:29.553 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: can't open /var/log/ntpstats/peerstats.20231101: Permission denied 2023-11-01T04:12:29.554 INFO:teuthology.orchestra.run.smithi181.stdout: 1 Nov 04:12:29 ntpd[27976]: ntpd: time slew -0.000827 s 2023-11-01T04:12:29.554 INFO:teuthology.orchestra.run.smithi181.stdout:ntpd: time slew -0.000827s 2023-11-01T04:12:29.554 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-11-01T04:12:29.555 INFO:teuthology.orchestra.run.smithi181.stderr: 1 Nov 04:12:29 ntpd[27976]: can't open /var/log/ntpstats/loopstats.20231101: Permission denied 2023-11-01T04:12:29.619 INFO:teuthology.orchestra.run.smithi181.stdout: remote refid st t when poll reach delay offset jitter 2023-11-01T04:12:29.619 INFO:teuthology.orchestra.run.smithi181.stdout:============================================================================== 2023-11-01T04:12:29.619 INFO:teuthology.orchestra.run.smithi181.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-01T04:12:29.620 INFO:teuthology.orchestra.run.smithi181.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-01T04:12:29.620 INFO:teuthology.orchestra.run.smithi181.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-01T04:12:29.620 INFO:teuthology.orchestra.run.smithi181.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-11-01T04:12:29.621 INFO:teuthology.run_tasks:Running task install... 2023-11-01T04:12:29.630 DEBUG:teuthology.task.install:project ceph 2023-11-01T04:12:29.630 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189'}} 2023-11-01T04:12:29.631 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189'} 2023-11-01T04:12:29.631 INFO:teuthology.task.install:Using flavor: default 2023-11-01T04:12:29.636 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-01T04:12:29.636 INFO:teuthology.task.install:extra packages: [] 2023-11-01T04:12:29.637 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-key list | grep Ceph 2023-11-01T04:12:29.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo apt-key list | grep Ceph 2023-11-01T04:12:29.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo apt-key list | grep Ceph 2023-11-01T04:12:29.639 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-key list | grep Ceph 2023-11-01T04:12:29.700 INFO:teuthology.orchestra.run.smithi173.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-11-01T04:12:29.701 INFO:teuthology.orchestra.run.smithi190.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-11-01T04:12:29.701 INFO:teuthology.orchestra.run.smithi153.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-11-01T04:12:29.718 INFO:teuthology.orchestra.run.smithi181.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-11-01T04:12:29.729 INFO:teuthology.orchestra.run.smithi190.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-11-01T04:12:29.729 INFO:teuthology.orchestra.run.smithi190.stdout:uid [ unknown] Ceph.com (release key) 2023-11-01T04:12:29.730 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-11-01T04:12:29.730 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-01T04:12:29.735 INFO:teuthology.orchestra.run.smithi173.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-11-01T04:12:29.736 INFO:teuthology.orchestra.run.smithi173.stdout:uid [ unknown] Ceph.com (release key) 2023-11-01T04:12:29.736 INFO:teuthology.orchestra.run.smithi153.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-11-01T04:12:29.736 INFO:teuthology.orchestra.run.smithi153.stdout:uid [ unknown] Ceph.com (release key) 2023-11-01T04:12:29.736 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-11-01T04:12:29.737 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-01T04:12:29.738 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-11-01T04:12:29.738 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-01T04:12:29.744 INFO:teuthology.orchestra.run.smithi181.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-11-01T04:12:29.744 INFO:teuthology.orchestra.run.smithi181.stdout:uid [ unknown] Ceph.com (release key) 2023-11-01T04:12:29.745 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-11-01T04:12:29.745 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-01T04:12:29.893 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default/ 2023-11-01T04:12:29.894 INFO:teuthology.task.install.deb:Package version is 18.0.0-6776-g420cd129-1jammy 2023-11-01T04:12:29.899 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default/ 2023-11-01T04:12:29.899 INFO:teuthology.task.install.deb:Package version is 18.0.0-6776-g420cd129-1jammy 2023-11-01T04:12:29.904 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default/ 2023-11-01T04:12:29.904 INFO:teuthology.task.install.deb:Package version is 18.0.0-6776-g420cd129-1jammy 2023-11-01T04:12:29.908 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default/ 2023-11-01T04:12:29.909 INFO:teuthology.task.install.deb:Package version is 18.0.0-6776-g420cd129-1jammy 2023-11-01T04:12:30.023 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:12:30.023 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-11-01T04:12:30.029 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:12:30.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-11-01T04:12:30.032 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:12:30.032 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-11-01T04:12:30.035 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:12:30.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-11-01T04:12:30.038 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get update 2023-11-01T04:12:30.049 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2023-11-01T04:12:30.051 DEBUG:teuthology.orchestra.run.smithi173:> sudo apt-get update 2023-11-01T04:12:30.082 DEBUG:teuthology.orchestra.run.smithi181:> sudo apt-get update 2023-11-01T04:12:30.204 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-01T04:12:30.213 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-01T04:12:30.219 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-01T04:12:30.219 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-01T04:12:30.230 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-01T04:12:30.231 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-01T04:12:30.244 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-01T04:12:30.244 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-01T04:12:30.250 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-01T04:12:30.254 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-01T04:12:30.257 INFO:teuthology.orchestra.run.smithi190.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy InRelease 2023-11-01T04:12:30.271 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-01T04:12:30.273 INFO:teuthology.orchestra.run.smithi153.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy InRelease 2023-11-01T04:12:30.279 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-11-01T04:12:30.281 INFO:teuthology.orchestra.run.smithi173.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy InRelease 2023-11-01T04:12:30.295 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-11-01T04:12:30.301 INFO:teuthology.orchestra.run.smithi190.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy Release.gpg 2023-11-01T04:12:30.305 INFO:teuthology.orchestra.run.smithi173.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-11-01T04:12:30.313 INFO:teuthology.orchestra.run.smithi181.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy InRelease 2023-11-01T04:12:30.319 INFO:teuthology.orchestra.run.smithi153.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy Release.gpg 2023-11-01T04:12:30.322 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,409 B] 2023-11-01T04:12:30.330 INFO:teuthology.orchestra.run.smithi173.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy Release.gpg 2023-11-01T04:12:30.337 INFO:teuthology.orchestra.run.smithi181.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-11-01T04:12:30.344 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-11-01T04:12:30.344 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,409 B] 2023-11-01T04:12:30.346 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-01T04:12:30.352 INFO:teuthology.orchestra.run.smithi190.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-11-01T04:12:30.354 INFO:teuthology.orchestra.run.smithi173.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-11-01T04:12:30.354 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-11-01T04:12:30.362 INFO:teuthology.orchestra.run.smithi181.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy Release.gpg 2023-11-01T04:12:30.379 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-11-01T04:12:30.383 INFO:teuthology.orchestra.run.smithi173.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,409 B] 2023-11-01T04:12:30.387 INFO:teuthology.orchestra.run.smithi181.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,409 B] 2023-11-01T04:12:30.416 INFO:teuthology.orchestra.run.smithi181.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-11-01T04:12:30.417 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-11-01T04:12:30.498 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-11-01T04:12:30.551 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 28.5 kB in 0s (84.4 kB/s) 2023-11-01T04:12:30.565 INFO:teuthology.orchestra.run.smithi173.stdout:Fetched 28.5 kB in 0s (84.5 kB/s) 2023-11-01T04:12:30.578 INFO:teuthology.orchestra.run.smithi181.stdout:Fetched 28.5 kB in 0s (88.6 kB/s) 2023-11-01T04:12:30.699 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 28.5 kB in 0s (60.2 kB/s) 2023-11-01T04:12:31.409 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2023-11-01T04:12:31.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6776-g420cd129-1jammy cephadm=18.0.0-6776-g420cd129-1jammy ceph-mds=18.0.0-6776-g420cd129-1jammy ceph-mgr=18.0.0-6776-g420cd129-1jammy ceph-common=18.0.0-6776-g420cd129-1jammy ceph-fuse=18.0.0-6776-g420cd129-1jammy ceph-test=18.0.0-6776-g420cd129-1jammy ceph-volume=18.0.0-6776-g420cd129-1jammy radosgw=18.0.0-6776-g420cd129-1jammy python3-rados=18.0.0-6776-g420cd129-1jammy python3-rgw=18.0.0-6776-g420cd129-1jammy python3-cephfs=18.0.0-6776-g420cd129-1jammy python3-rbd=18.0.0-6776-g420cd129-1jammy libcephfs2=18.0.0-6776-g420cd129-1jammy libcephfs-dev=18.0.0-6776-g420cd129-1jammy librados2=18.0.0-6776-g420cd129-1jammy librbd1=18.0.0-6776-g420cd129-1jammy rbd-fuse=18.0.0-6776-g420cd129-1jammy 2023-11-01T04:12:31.433 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2023-11-01T04:12:31.447 DEBUG:teuthology.orchestra.run.smithi173:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6776-g420cd129-1jammy cephadm=18.0.0-6776-g420cd129-1jammy ceph-mds=18.0.0-6776-g420cd129-1jammy ceph-mgr=18.0.0-6776-g420cd129-1jammy ceph-common=18.0.0-6776-g420cd129-1jammy ceph-fuse=18.0.0-6776-g420cd129-1jammy ceph-test=18.0.0-6776-g420cd129-1jammy ceph-volume=18.0.0-6776-g420cd129-1jammy radosgw=18.0.0-6776-g420cd129-1jammy python3-rados=18.0.0-6776-g420cd129-1jammy python3-rgw=18.0.0-6776-g420cd129-1jammy python3-cephfs=18.0.0-6776-g420cd129-1jammy python3-rbd=18.0.0-6776-g420cd129-1jammy libcephfs2=18.0.0-6776-g420cd129-1jammy libcephfs-dev=18.0.0-6776-g420cd129-1jammy librados2=18.0.0-6776-g420cd129-1jammy librbd1=18.0.0-6776-g420cd129-1jammy rbd-fuse=18.0.0-6776-g420cd129-1jammy 2023-11-01T04:12:31.474 INFO:teuthology.orchestra.run.smithi181.stdout:Reading package lists... 2023-11-01T04:12:31.483 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2023-11-01T04:12:31.489 DEBUG:teuthology.orchestra.run.smithi181:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6776-g420cd129-1jammy cephadm=18.0.0-6776-g420cd129-1jammy ceph-mds=18.0.0-6776-g420cd129-1jammy ceph-mgr=18.0.0-6776-g420cd129-1jammy ceph-common=18.0.0-6776-g420cd129-1jammy ceph-fuse=18.0.0-6776-g420cd129-1jammy ceph-test=18.0.0-6776-g420cd129-1jammy ceph-volume=18.0.0-6776-g420cd129-1jammy radosgw=18.0.0-6776-g420cd129-1jammy python3-rados=18.0.0-6776-g420cd129-1jammy python3-rgw=18.0.0-6776-g420cd129-1jammy python3-cephfs=18.0.0-6776-g420cd129-1jammy python3-rbd=18.0.0-6776-g420cd129-1jammy libcephfs2=18.0.0-6776-g420cd129-1jammy libcephfs-dev=18.0.0-6776-g420cd129-1jammy librados2=18.0.0-6776-g420cd129-1jammy librbd1=18.0.0-6776-g420cd129-1jammy rbd-fuse=18.0.0-6776-g420cd129-1jammy 2023-11-01T04:12:31.506 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2023-11-01T04:12:31.550 INFO:teuthology.orchestra.run.smithi181.stdout:Reading package lists... 2023-11-01T04:12:31.569 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2023-11-01T04:12:31.584 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6776-g420cd129-1jammy cephadm=18.0.0-6776-g420cd129-1jammy ceph-mds=18.0.0-6776-g420cd129-1jammy ceph-mgr=18.0.0-6776-g420cd129-1jammy ceph-common=18.0.0-6776-g420cd129-1jammy ceph-fuse=18.0.0-6776-g420cd129-1jammy ceph-test=18.0.0-6776-g420cd129-1jammy ceph-volume=18.0.0-6776-g420cd129-1jammy radosgw=18.0.0-6776-g420cd129-1jammy python3-rados=18.0.0-6776-g420cd129-1jammy python3-rgw=18.0.0-6776-g420cd129-1jammy python3-cephfs=18.0.0-6776-g420cd129-1jammy python3-rbd=18.0.0-6776-g420cd129-1jammy libcephfs2=18.0.0-6776-g420cd129-1jammy libcephfs-dev=18.0.0-6776-g420cd129-1jammy librados2=18.0.0-6776-g420cd129-1jammy librbd1=18.0.0-6776-g420cd129-1jammy rbd-fuse=18.0.0-6776-g420cd129-1jammy 2023-11-01T04:12:31.645 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2023-11-01T04:12:31.675 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2023-11-01T04:12:31.675 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2023-11-01T04:12:31.693 INFO:teuthology.orchestra.run.smithi173.stdout:Building dependency tree... 2023-11-01T04:12:31.693 INFO:teuthology.orchestra.run.smithi173.stdout:Reading state information... 2023-11-01T04:12:31.736 INFO:teuthology.orchestra.run.smithi181.stdout:Building dependency tree... 2023-11-01T04:12:31.737 INFO:teuthology.orchestra.run.smithi181.stdout:Reading state information... 2023-11-01T04:12:31.830 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2023-11-01T04:12:31.831 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2023-11-01T04:12:31.842 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages were automatically installed and are no longer required: 2023-11-01T04:12:31.843 INFO:teuthology.orchestra.run.smithi190.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-01T04:12:31.843 INFO:teuthology.orchestra.run.smithi190.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-01T04:12:31.843 INFO:teuthology.orchestra.run.smithi190.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-01T04:12:31.844 INFO:teuthology.orchestra.run.smithi190.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-01T04:12:31.844 INFO:teuthology.orchestra.run.smithi190.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-01T04:12:31.844 INFO:teuthology.orchestra.run.smithi190.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-01T04:12:31.845 INFO:teuthology.orchestra.run.smithi190.stdout:The following additional packages will be installed: 2023-11-01T04:12:31.845 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-11-01T04:12:31.845 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-11-01T04:12:31.845 INFO:teuthology.orchestra.run.smithi190.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-11-01T04:12:31.846 INFO:teuthology.orchestra.run.smithi190.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-11-01T04:12:31.847 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-11-01T04:12:31.847 INFO:teuthology.orchestra.run.smithi190.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-11-01T04:12:31.847 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-11-01T04:12:31.847 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-11-01T04:12:31.848 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-11-01T04:12:31.848 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-11-01T04:12:31.848 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-11-01T04:12:31.848 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-11-01T04:12:31.848 INFO:teuthology.orchestra.run.smithi190.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-11-01T04:12:31.848 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-11-01T04:12:31.849 INFO:teuthology.orchestra.run.smithi190.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-11-01T04:12:31.849 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-11-01T04:12:31.849 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-11-01T04:12:31.849 INFO:teuthology.orchestra.run.smithi190.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-01T04:12:31.850 INFO:teuthology.orchestra.run.smithi190.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-01T04:12:31.850 INFO:teuthology.orchestra.run.smithi190.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-01T04:12:31.850 INFO:teuthology.orchestra.run.smithi190.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-01T04:12:31.850 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-11-01T04:12:31.850 INFO:teuthology.orchestra.run.smithi190.stdout:Suggested packages: 2023-11-01T04:12:31.851 INFO:teuthology.orchestra.run.smithi190.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-11-01T04:12:31.851 INFO:teuthology.orchestra.run.smithi190.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-11-01T04:12:31.851 INFO:teuthology.orchestra.run.smithi190.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-11-01T04:12:31.851 INFO:teuthology.orchestra.run.smithi190.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-11-01T04:12:31.851 INFO:teuthology.orchestra.run.smithi190.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-11-01T04:12:31.852 INFO:teuthology.orchestra.run.smithi190.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-11-01T04:12:31.852 INFO:teuthology.orchestra.run.smithi190.stdout: python3-watchdog 2023-11-01T04:12:31.852 INFO:teuthology.orchestra.run.smithi190.stdout:Recommended packages: 2023-11-01T04:12:31.852 INFO:teuthology.orchestra.run.smithi190.stdout: btrfs-tools 2023-11-01T04:12:31.861 INFO:teuthology.orchestra.run.smithi173.stdout:The following packages were automatically installed and are no longer required: 2023-11-01T04:12:31.862 INFO:teuthology.orchestra.run.smithi173.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-01T04:12:31.862 INFO:teuthology.orchestra.run.smithi173.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-01T04:12:31.862 INFO:teuthology.orchestra.run.smithi173.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-01T04:12:31.862 INFO:teuthology.orchestra.run.smithi173.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-01T04:12:31.864 INFO:teuthology.orchestra.run.smithi173.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-01T04:12:31.864 INFO:teuthology.orchestra.run.smithi173.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-01T04:12:31.864 INFO:teuthology.orchestra.run.smithi173.stdout:The following additional packages will be installed: 2023-11-01T04:12:31.865 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-11-01T04:12:31.865 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-11-01T04:12:31.865 INFO:teuthology.orchestra.run.smithi173.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-11-01T04:12:31.865 INFO:teuthology.orchestra.run.smithi173.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-11-01T04:12:31.865 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-11-01T04:12:31.866 INFO:teuthology.orchestra.run.smithi173.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-11-01T04:12:31.866 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-11-01T04:12:31.866 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-11-01T04:12:31.866 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-11-01T04:12:31.867 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-11-01T04:12:31.867 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-11-01T04:12:31.867 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-11-01T04:12:31.867 INFO:teuthology.orchestra.run.smithi173.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-11-01T04:12:31.867 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-11-01T04:12:31.868 INFO:teuthology.orchestra.run.smithi173.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-11-01T04:12:31.868 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-11-01T04:12:31.868 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-11-01T04:12:31.868 INFO:teuthology.orchestra.run.smithi173.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-01T04:12:31.868 INFO:teuthology.orchestra.run.smithi173.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-01T04:12:31.869 INFO:teuthology.orchestra.run.smithi173.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-01T04:12:31.869 INFO:teuthology.orchestra.run.smithi173.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-01T04:12:31.869 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-11-01T04:12:31.870 INFO:teuthology.orchestra.run.smithi173.stdout:Suggested packages: 2023-11-01T04:12:31.870 INFO:teuthology.orchestra.run.smithi173.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-11-01T04:12:31.870 INFO:teuthology.orchestra.run.smithi173.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-11-01T04:12:31.870 INFO:teuthology.orchestra.run.smithi173.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-11-01T04:12:31.871 INFO:teuthology.orchestra.run.smithi173.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-11-01T04:12:31.871 INFO:teuthology.orchestra.run.smithi173.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-11-01T04:12:31.871 INFO:teuthology.orchestra.run.smithi173.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-11-01T04:12:31.871 INFO:teuthology.orchestra.run.smithi173.stdout: python3-watchdog 2023-11-01T04:12:31.871 INFO:teuthology.orchestra.run.smithi173.stdout:Recommended packages: 2023-11-01T04:12:31.872 INFO:teuthology.orchestra.run.smithi173.stdout: btrfs-tools 2023-11-01T04:12:31.897 INFO:teuthology.orchestra.run.smithi181.stdout:The following packages were automatically installed and are no longer required: 2023-11-01T04:12:31.897 INFO:teuthology.orchestra.run.smithi181.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-01T04:12:31.897 INFO:teuthology.orchestra.run.smithi181.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-01T04:12:31.897 INFO:teuthology.orchestra.run.smithi181.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-01T04:12:31.898 INFO:teuthology.orchestra.run.smithi181.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-01T04:12:31.899 INFO:teuthology.orchestra.run.smithi181.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-01T04:12:31.899 INFO:teuthology.orchestra.run.smithi181.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-01T04:12:31.900 INFO:teuthology.orchestra.run.smithi181.stdout:The following additional packages will be installed: 2023-11-01T04:12:31.900 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-11-01T04:12:31.900 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-11-01T04:12:31.900 INFO:teuthology.orchestra.run.smithi181.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-11-01T04:12:31.900 INFO:teuthology.orchestra.run.smithi181.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-11-01T04:12:31.901 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-11-01T04:12:31.901 INFO:teuthology.orchestra.run.smithi181.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-11-01T04:12:31.901 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-11-01T04:12:31.901 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-11-01T04:12:31.901 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-11-01T04:12:31.902 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-11-01T04:12:31.902 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-11-01T04:12:31.902 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-11-01T04:12:31.902 INFO:teuthology.orchestra.run.smithi181.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-11-01T04:12:31.903 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-11-01T04:12:31.903 INFO:teuthology.orchestra.run.smithi181.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-11-01T04:12:31.903 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-11-01T04:12:31.903 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-11-01T04:12:31.903 INFO:teuthology.orchestra.run.smithi181.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-01T04:12:31.903 INFO:teuthology.orchestra.run.smithi181.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-01T04:12:31.904 INFO:teuthology.orchestra.run.smithi181.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-01T04:12:31.904 INFO:teuthology.orchestra.run.smithi181.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-01T04:12:31.904 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-11-01T04:12:31.905 INFO:teuthology.orchestra.run.smithi181.stdout:Suggested packages: 2023-11-01T04:12:31.905 INFO:teuthology.orchestra.run.smithi181.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-11-01T04:12:31.905 INFO:teuthology.orchestra.run.smithi181.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-11-01T04:12:31.905 INFO:teuthology.orchestra.run.smithi181.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-11-01T04:12:31.906 INFO:teuthology.orchestra.run.smithi181.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-11-01T04:12:31.906 INFO:teuthology.orchestra.run.smithi181.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-11-01T04:12:31.906 INFO:teuthology.orchestra.run.smithi181.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-11-01T04:12:31.906 INFO:teuthology.orchestra.run.smithi181.stdout: python3-watchdog 2023-11-01T04:12:31.906 INFO:teuthology.orchestra.run.smithi181.stdout:Recommended packages: 2023-11-01T04:12:31.907 INFO:teuthology.orchestra.run.smithi181.stdout: btrfs-tools 2023-11-01T04:12:31.942 INFO:teuthology.orchestra.run.smithi190.stdout:The following NEW packages will be installed: 2023-11-01T04:12:31.943 INFO:teuthology.orchestra.run.smithi190.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-11-01T04:12:31.943 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-11-01T04:12:31.943 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-11-01T04:12:31.944 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-11-01T04:12:31.945 INFO:teuthology.orchestra.run.smithi190.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-11-01T04:12:31.945 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-11-01T04:12:31.945 INFO:teuthology.orchestra.run.smithi190.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-11-01T04:12:31.945 INFO:teuthology.orchestra.run.smithi190.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-11-01T04:12:31.946 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-11-01T04:12:31.946 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-11-01T04:12:31.946 INFO:teuthology.orchestra.run.smithi190.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-11-01T04:12:31.946 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-11-01T04:12:31.946 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-11-01T04:12:31.947 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-11-01T04:12:31.947 INFO:teuthology.orchestra.run.smithi190.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-11-01T04:12:31.947 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-11-01T04:12:31.947 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-11-01T04:12:31.947 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-11-01T04:12:31.948 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-11-01T04:12:31.948 INFO:teuthology.orchestra.run.smithi190.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-01T04:12:31.948 INFO:teuthology.orchestra.run.smithi190.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-01T04:12:31.948 INFO:teuthology.orchestra.run.smithi190.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-01T04:12:31.948 INFO:teuthology.orchestra.run.smithi190.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-01T04:12:31.948 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-11-01T04:12:31.949 INFO:teuthology.orchestra.run.smithi190.stdout: socat xmlstarlet 2023-11-01T04:12:31.961 INFO:teuthology.orchestra.run.smithi173.stdout:The following NEW packages will be installed: 2023-11-01T04:12:31.961 INFO:teuthology.orchestra.run.smithi173.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-11-01T04:12:31.961 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-11-01T04:12:31.961 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-11-01T04:12:31.962 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-11-01T04:12:31.962 INFO:teuthology.orchestra.run.smithi173.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-11-01T04:12:31.962 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-11-01T04:12:31.963 INFO:teuthology.orchestra.run.smithi173.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-11-01T04:12:31.963 INFO:teuthology.orchestra.run.smithi173.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-11-01T04:12:31.964 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-11-01T04:12:31.964 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-11-01T04:12:31.964 INFO:teuthology.orchestra.run.smithi173.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-11-01T04:12:31.964 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-11-01T04:12:31.964 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-11-01T04:12:31.965 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-11-01T04:12:31.965 INFO:teuthology.orchestra.run.smithi173.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-11-01T04:12:31.965 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-11-01T04:12:31.965 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-11-01T04:12:31.965 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-11-01T04:12:31.965 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-11-01T04:12:31.966 INFO:teuthology.orchestra.run.smithi173.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-01T04:12:31.966 INFO:teuthology.orchestra.run.smithi173.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-01T04:12:31.966 INFO:teuthology.orchestra.run.smithi173.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-01T04:12:31.966 INFO:teuthology.orchestra.run.smithi173.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-01T04:12:31.967 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-11-01T04:12:31.967 INFO:teuthology.orchestra.run.smithi173.stdout: socat xmlstarlet 2023-11-01T04:12:31.995 INFO:teuthology.orchestra.run.smithi181.stdout:The following NEW packages will be installed: 2023-11-01T04:12:31.995 INFO:teuthology.orchestra.run.smithi181.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-11-01T04:12:31.995 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-11-01T04:12:31.996 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-11-01T04:12:31.997 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-11-01T04:12:31.997 INFO:teuthology.orchestra.run.smithi181.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-11-01T04:12:31.997 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-11-01T04:12:31.998 INFO:teuthology.orchestra.run.smithi181.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-11-01T04:12:31.998 INFO:teuthology.orchestra.run.smithi181.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-11-01T04:12:31.998 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-11-01T04:12:31.998 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-11-01T04:12:31.998 INFO:teuthology.orchestra.run.smithi181.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-11-01T04:12:31.999 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-11-01T04:12:31.999 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-11-01T04:12:31.999 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-11-01T04:12:31.999 INFO:teuthology.orchestra.run.smithi181.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-11-01T04:12:32.000 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-11-01T04:12:32.000 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-11-01T04:12:32.000 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-11-01T04:12:32.000 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-11-01T04:12:32.000 INFO:teuthology.orchestra.run.smithi181.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-01T04:12:32.001 INFO:teuthology.orchestra.run.smithi181.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-01T04:12:32.001 INFO:teuthology.orchestra.run.smithi181.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-01T04:12:32.001 INFO:teuthology.orchestra.run.smithi181.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-01T04:12:32.001 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-11-01T04:12:32.002 INFO:teuthology.orchestra.run.smithi181.stdout: socat xmlstarlet 2023-11-01T04:12:32.003 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages were automatically installed and are no longer required: 2023-11-01T04:12:32.004 INFO:teuthology.orchestra.run.smithi153.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-11-01T04:12:32.004 INFO:teuthology.orchestra.run.smithi153.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-11-01T04:12:32.004 INFO:teuthology.orchestra.run.smithi153.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-11-01T04:12:32.004 INFO:teuthology.orchestra.run.smithi153.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-11-01T04:12:32.005 INFO:teuthology.orchestra.run.smithi153.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-11-01T04:12:32.005 INFO:teuthology.orchestra.run.smithi153.stdout:Use 'sudo apt autoremove' to remove them. 2023-11-01T04:12:32.005 INFO:teuthology.orchestra.run.smithi153.stdout:The following additional packages will be installed: 2023-11-01T04:12:32.005 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-11-01T04:12:32.006 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-11-01T04:12:32.006 INFO:teuthology.orchestra.run.smithi153.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-11-01T04:12:32.006 INFO:teuthology.orchestra.run.smithi153.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-11-01T04:12:32.006 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-11-01T04:12:32.006 INFO:teuthology.orchestra.run.smithi153.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-11-01T04:12:32.007 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-11-01T04:12:32.007 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-11-01T04:12:32.007 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-11-01T04:12:32.007 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-11-01T04:12:32.007 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-11-01T04:12:32.008 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-11-01T04:12:32.008 INFO:teuthology.orchestra.run.smithi153.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-11-01T04:12:32.008 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-11-01T04:12:32.008 INFO:teuthology.orchestra.run.smithi153.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-11-01T04:12:32.008 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-11-01T04:12:32.009 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-11-01T04:12:32.009 INFO:teuthology.orchestra.run.smithi153.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-01T04:12:32.009 INFO:teuthology.orchestra.run.smithi153.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-01T04:12:32.009 INFO:teuthology.orchestra.run.smithi153.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-01T04:12:32.009 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-01T04:12:32.010 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-11-01T04:12:32.010 INFO:teuthology.orchestra.run.smithi153.stdout:Suggested packages: 2023-11-01T04:12:32.010 INFO:teuthology.orchestra.run.smithi153.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-11-01T04:12:32.010 INFO:teuthology.orchestra.run.smithi153.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-11-01T04:12:32.011 INFO:teuthology.orchestra.run.smithi153.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-11-01T04:12:32.011 INFO:teuthology.orchestra.run.smithi153.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-11-01T04:12:32.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-11-01T04:12:32.011 INFO:teuthology.orchestra.run.smithi153.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-11-01T04:12:32.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-watchdog 2023-11-01T04:12:32.012 INFO:teuthology.orchestra.run.smithi153.stdout:Recommended packages: 2023-11-01T04:12:32.012 INFO:teuthology.orchestra.run.smithi153.stdout: btrfs-tools 2023-11-01T04:12:32.017 INFO:teuthology.orchestra.run.smithi173.stdout:0 upgraded, 109 newly installed, 0 to remove and 257 not upgraded. 2023-11-01T04:12:32.018 INFO:teuthology.orchestra.run.smithi173.stdout:Need to get 184 MB of archives. 2023-11-01T04:12:32.018 INFO:teuthology.orchestra.run.smithi173.stdout:After this operation, 825 MB of additional disk space will be used. 2023-11-01T04:12:32.018 INFO:teuthology.orchestra.run.smithi173.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-11-01T04:12:32.053 INFO:teuthology.orchestra.run.smithi181.stdout:0 upgraded, 109 newly installed, 0 to remove and 257 not upgraded. 2023-11-01T04:12:32.053 INFO:teuthology.orchestra.run.smithi181.stdout:Need to get 184 MB of archives. 2023-11-01T04:12:32.053 INFO:teuthology.orchestra.run.smithi181.stdout:After this operation, 825 MB of additional disk space will be used. 2023-11-01T04:12:32.053 INFO:teuthology.orchestra.run.smithi181.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-11-01T04:12:32.074 INFO:teuthology.orchestra.run.smithi173.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-11-01T04:12:32.097 INFO:teuthology.orchestra.run.smithi153.stdout:The following NEW packages will be installed: 2023-11-01T04:12:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-11-01T04:12:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-11-01T04:12:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-11-01T04:12:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-11-01T04:12:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-11-01T04:12:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-11-01T04:12:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-11-01T04:12:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-11-01T04:12:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-11-01T04:12:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-11-01T04:12:32.102 INFO:teuthology.orchestra.run.smithi153.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-11-01T04:12:32.102 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-11-01T04:12:32.102 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-11-01T04:12:32.102 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-11-01T04:12:32.102 INFO:teuthology.orchestra.run.smithi153.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-11-01T04:12:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-11-01T04:12:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-11-01T04:12:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-11-01T04:12:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-11-01T04:12:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-11-01T04:12:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-11-01T04:12:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-11-01T04:12:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-11-01T04:12:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-11-01T04:12:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: socat xmlstarlet 2023-11-01T04:12:32.105 INFO:teuthology.orchestra.run.smithi173.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6776-g420cd129-1jammy [3,449 kB] 2023-11-01T04:12:32.105 INFO:teuthology.orchestra.run.smithi190.stdout:0 upgraded, 109 newly installed, 0 to remove and 257 not upgraded. 2023-11-01T04:12:32.106 INFO:teuthology.orchestra.run.smithi190.stdout:Need to get 184 MB of archives. 2023-11-01T04:12:32.106 INFO:teuthology.orchestra.run.smithi190.stdout:After this operation, 825 MB of additional disk space will be used. 2023-11-01T04:12:32.106 INFO:teuthology.orchestra.run.smithi190.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6776-g420cd129-1jammy [3,449 kB] 2023-11-01T04:12:32.110 INFO:teuthology.orchestra.run.smithi173.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-11-01T04:12:32.113 INFO:teuthology.orchestra.run.smithi173.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-11-01T04:12:32.113 INFO:teuthology.orchestra.run.smithi181.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-11-01T04:12:32.128 INFO:teuthology.orchestra.run.smithi173.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-11-01T04:12:32.131 INFO:teuthology.orchestra.run.smithi190.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-11-01T04:12:32.138 INFO:teuthology.orchestra.run.smithi181.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6776-g420cd129-1jammy [3,449 kB] 2023-11-01T04:12:32.149 INFO:teuthology.orchestra.run.smithi181.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-11-01T04:12:32.152 INFO:teuthology.orchestra.run.smithi181.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-11-01T04:12:32.156 INFO:teuthology.orchestra.run.smithi153.stdout:0 upgraded, 109 newly installed, 0 to remove and 257 not upgraded. 2023-11-01T04:12:32.156 INFO:teuthology.orchestra.run.smithi153.stdout:Need to get 184 MB of archives. 2023-11-01T04:12:32.156 INFO:teuthology.orchestra.run.smithi153.stdout:After this operation, 825 MB of additional disk space will be used. 2023-11-01T04:12:32.157 INFO:teuthology.orchestra.run.smithi153.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-11-01T04:12:32.166 INFO:teuthology.orchestra.run.smithi181.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-11-01T04:12:32.178 INFO:teuthology.orchestra.run.smithi173.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-11-01T04:12:32.182 INFO:teuthology.orchestra.run.smithi173.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-11-01T04:12:32.193 INFO:teuthology.orchestra.run.smithi173.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-11-01T04:12:32.197 INFO:teuthology.orchestra.run.smithi173.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-11-01T04:12:32.198 INFO:teuthology.orchestra.run.smithi173.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-11-01T04:12:32.199 INFO:teuthology.orchestra.run.smithi173.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-11-01T04:12:32.206 INFO:teuthology.orchestra.run.smithi173.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-11-01T04:12:32.207 INFO:teuthology.orchestra.run.smithi173.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-11-01T04:12:32.207 INFO:teuthology.orchestra.run.smithi173.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-11-01T04:12:32.208 INFO:teuthology.orchestra.run.smithi173.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-11-01T04:12:32.213 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-11-01T04:12:32.215 INFO:teuthology.orchestra.run.smithi173.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-11-01T04:12:32.216 INFO:teuthology.orchestra.run.smithi173.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-11-01T04:12:32.217 INFO:teuthology.orchestra.run.smithi181.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-11-01T04:12:32.222 INFO:teuthology.orchestra.run.smithi181.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-11-01T04:12:32.232 INFO:teuthology.orchestra.run.smithi173.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-11-01T04:12:32.233 INFO:teuthology.orchestra.run.smithi173.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-11-01T04:12:32.233 INFO:teuthology.orchestra.run.smithi173.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-11-01T04:12:32.233 INFO:teuthology.orchestra.run.smithi173.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-11-01T04:12:32.234 INFO:teuthology.orchestra.run.smithi181.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-11-01T04:12:32.236 INFO:teuthology.orchestra.run.smithi173.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-11-01T04:12:32.236 INFO:teuthology.orchestra.run.smithi173.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-11-01T04:12:32.237 INFO:teuthology.orchestra.run.smithi173.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-11-01T04:12:32.238 INFO:teuthology.orchestra.run.smithi181.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-11-01T04:12:32.238 INFO:teuthology.orchestra.run.smithi173.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-11-01T04:12:32.238 INFO:teuthology.orchestra.run.smithi181.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-11-01T04:12:32.239 INFO:teuthology.orchestra.run.smithi181.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-11-01T04:12:32.247 INFO:teuthology.orchestra.run.smithi181.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-11-01T04:12:32.247 INFO:teuthology.orchestra.run.smithi181.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-11-01T04:12:32.248 INFO:teuthology.orchestra.run.smithi181.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-11-01T04:12:32.248 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-11-01T04:12:32.249 INFO:teuthology.orchestra.run.smithi181.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-11-01T04:12:32.249 INFO:teuthology.orchestra.run.smithi173.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-11-01T04:12:32.250 INFO:teuthology.orchestra.run.smithi173.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-11-01T04:12:32.251 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-11-01T04:12:32.255 INFO:teuthology.orchestra.run.smithi181.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-11-01T04:12:32.256 INFO:teuthology.orchestra.run.smithi181.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-11-01T04:12:32.266 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-11-01T04:12:32.266 INFO:teuthology.orchestra.run.smithi173.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-11-01T04:12:32.268 INFO:teuthology.orchestra.run.smithi173.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-11-01T04:12:32.269 INFO:teuthology.orchestra.run.smithi173.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-11-01T04:12:32.269 INFO:teuthology.orchestra.run.smithi173.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-11-01T04:12:32.272 INFO:teuthology.orchestra.run.smithi181.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-11-01T04:12:32.273 INFO:teuthology.orchestra.run.smithi181.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-11-01T04:12:32.273 INFO:teuthology.orchestra.run.smithi181.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-11-01T04:12:32.273 INFO:teuthology.orchestra.run.smithi181.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-11-01T04:12:32.274 INFO:teuthology.orchestra.run.smithi173.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-11-01T04:12:32.274 INFO:teuthology.orchestra.run.smithi173.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-11-01T04:12:32.275 INFO:teuthology.orchestra.run.smithi173.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-11-01T04:12:32.276 INFO:teuthology.orchestra.run.smithi181.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-11-01T04:12:32.276 INFO:teuthology.orchestra.run.smithi173.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-11-01T04:12:32.276 INFO:teuthology.orchestra.run.smithi181.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-11-01T04:12:32.277 INFO:teuthology.orchestra.run.smithi181.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-11-01T04:12:32.277 INFO:teuthology.orchestra.run.smithi181.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-11-01T04:12:32.283 INFO:teuthology.orchestra.run.smithi173.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-11-01T04:12:32.285 INFO:teuthology.orchestra.run.smithi173.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-11-01T04:12:32.289 INFO:teuthology.orchestra.run.smithi181.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-11-01T04:12:32.289 INFO:teuthology.orchestra.run.smithi181.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-11-01T04:12:32.294 INFO:teuthology.orchestra.run.smithi173.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-11-01T04:12:32.295 INFO:teuthology.orchestra.run.smithi173.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-11-01T04:12:32.296 INFO:teuthology.orchestra.run.smithi173.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-11-01T04:12:32.299 INFO:teuthology.orchestra.run.smithi173.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-11-01T04:12:32.299 INFO:teuthology.orchestra.run.smithi173.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-11-01T04:12:32.302 INFO:teuthology.orchestra.run.smithi173.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-11-01T04:12:32.306 INFO:teuthology.orchestra.run.smithi181.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-11-01T04:12:32.307 INFO:teuthology.orchestra.run.smithi181.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-11-01T04:12:32.308 INFO:teuthology.orchestra.run.smithi181.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-11-01T04:12:32.308 INFO:teuthology.orchestra.run.smithi181.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-11-01T04:12:32.314 INFO:teuthology.orchestra.run.smithi181.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-11-01T04:12:32.314 INFO:teuthology.orchestra.run.smithi181.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-11-01T04:12:32.314 INFO:teuthology.orchestra.run.smithi181.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-11-01T04:12:32.315 INFO:teuthology.orchestra.run.smithi181.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-11-01T04:12:32.317 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-11-01T04:12:32.321 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-11-01T04:12:32.322 INFO:teuthology.orchestra.run.smithi181.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-11-01T04:12:32.324 INFO:teuthology.orchestra.run.smithi181.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-11-01T04:12:32.332 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-11-01T04:12:32.333 INFO:teuthology.orchestra.run.smithi181.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-11-01T04:12:32.335 INFO:teuthology.orchestra.run.smithi181.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-11-01T04:12:32.335 INFO:teuthology.orchestra.run.smithi181.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-11-01T04:12:32.336 INFO:teuthology.orchestra.run.smithi153.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-11-01T04:12:32.337 INFO:teuthology.orchestra.run.smithi153.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-11-01T04:12:32.337 INFO:teuthology.orchestra.run.smithi153.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-11-01T04:12:32.338 INFO:teuthology.orchestra.run.smithi181.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-11-01T04:12:32.338 INFO:teuthology.orchestra.run.smithi181.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-11-01T04:12:32.341 INFO:teuthology.orchestra.run.smithi181.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-11-01T04:12:32.345 INFO:teuthology.orchestra.run.smithi153.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-11-01T04:12:32.346 INFO:teuthology.orchestra.run.smithi153.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-11-01T04:12:32.347 INFO:teuthology.orchestra.run.smithi153.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-11-01T04:12:32.348 INFO:teuthology.orchestra.run.smithi153.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-11-01T04:12:32.353 INFO:teuthology.orchestra.run.smithi173.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-11-01T04:12:32.355 INFO:teuthology.orchestra.run.smithi173.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-11-01T04:12:32.355 INFO:teuthology.orchestra.run.smithi153.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-11-01T04:12:32.355 INFO:teuthology.orchestra.run.smithi153.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-11-01T04:12:32.356 INFO:teuthology.orchestra.run.smithi173.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-11-01T04:12:32.356 INFO:teuthology.orchestra.run.smithi173.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-11-01T04:12:32.357 INFO:teuthology.orchestra.run.smithi173.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-11-01T04:12:32.358 INFO:teuthology.orchestra.run.smithi173.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-11-01T04:12:32.371 INFO:teuthology.orchestra.run.smithi153.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-11-01T04:12:32.372 INFO:teuthology.orchestra.run.smithi153.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-11-01T04:12:32.372 INFO:teuthology.orchestra.run.smithi153.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-11-01T04:12:32.373 INFO:teuthology.orchestra.run.smithi153.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-11-01T04:12:32.374 INFO:teuthology.orchestra.run.smithi173.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-11-01T04:12:32.375 INFO:teuthology.orchestra.run.smithi153.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-11-01T04:12:32.376 INFO:teuthology.orchestra.run.smithi153.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-11-01T04:12:32.376 INFO:teuthology.orchestra.run.smithi173.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-11-01T04:12:32.376 INFO:teuthology.orchestra.run.smithi153.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-11-01T04:12:32.377 INFO:teuthology.orchestra.run.smithi173.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-11-01T04:12:32.377 INFO:teuthology.orchestra.run.smithi153.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-11-01T04:12:32.386 INFO:teuthology.orchestra.run.smithi153.stdout:Get:26 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6776-g420cd129-1jammy [3,449 kB] 2023-11-01T04:12:32.388 INFO:teuthology.orchestra.run.smithi153.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-11-01T04:12:32.388 INFO:teuthology.orchestra.run.smithi153.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-11-01T04:12:32.390 INFO:teuthology.orchestra.run.smithi173.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-11-01T04:12:32.391 INFO:teuthology.orchestra.run.smithi173.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-11-01T04:12:32.391 INFO:teuthology.orchestra.run.smithi173.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-11-01T04:12:32.392 INFO:teuthology.orchestra.run.smithi173.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-11-01T04:12:32.392 INFO:teuthology.orchestra.run.smithi181.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-11-01T04:12:32.393 INFO:teuthology.orchestra.run.smithi173.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-11-01T04:12:32.393 INFO:teuthology.orchestra.run.smithi181.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-11-01T04:12:32.393 INFO:teuthology.orchestra.run.smithi173.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-11-01T04:12:32.394 INFO:teuthology.orchestra.run.smithi181.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-11-01T04:12:32.395 INFO:teuthology.orchestra.run.smithi181.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-11-01T04:12:32.395 INFO:teuthology.orchestra.run.smithi181.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-11-01T04:12:32.405 INFO:teuthology.orchestra.run.smithi181.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-11-01T04:12:32.405 INFO:teuthology.orchestra.run.smithi153.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-11-01T04:12:32.406 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-11-01T04:12:32.406 INFO:teuthology.orchestra.run.smithi153.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-11-01T04:12:32.407 INFO:teuthology.orchestra.run.smithi173.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-11-01T04:12:32.408 INFO:teuthology.orchestra.run.smithi173.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-11-01T04:12:32.408 INFO:teuthology.orchestra.run.smithi153.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-11-01T04:12:32.408 INFO:teuthology.orchestra.run.smithi153.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-11-01T04:12:32.410 INFO:teuthology.orchestra.run.smithi173.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-11-01T04:12:32.413 INFO:teuthology.orchestra.run.smithi153.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-11-01T04:12:32.413 INFO:teuthology.orchestra.run.smithi153.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-11-01T04:12:32.413 INFO:teuthology.orchestra.run.smithi153.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-11-01T04:12:32.414 INFO:teuthology.orchestra.run.smithi153.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-11-01T04:12:32.422 INFO:teuthology.orchestra.run.smithi153.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-11-01T04:12:32.422 INFO:teuthology.orchestra.run.smithi181.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-11-01T04:12:32.424 INFO:teuthology.orchestra.run.smithi173.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-11-01T04:12:32.424 INFO:teuthology.orchestra.run.smithi153.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-11-01T04:12:32.425 INFO:teuthology.orchestra.run.smithi181.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-11-01T04:12:32.425 INFO:teuthology.orchestra.run.smithi181.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-11-01T04:12:32.425 INFO:teuthology.orchestra.run.smithi173.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-11-01T04:12:32.426 INFO:teuthology.orchestra.run.smithi173.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-11-01T04:12:32.428 INFO:teuthology.orchestra.run.smithi173.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-11-01T04:12:32.431 INFO:teuthology.orchestra.run.smithi173.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-11-01T04:12:32.433 INFO:teuthology.orchestra.run.smithi153.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-11-01T04:12:32.434 INFO:teuthology.orchestra.run.smithi153.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-11-01T04:12:32.435 INFO:teuthology.orchestra.run.smithi173.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-11-01T04:12:32.435 INFO:teuthology.orchestra.run.smithi153.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-11-01T04:12:32.437 INFO:teuthology.orchestra.run.smithi173.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-11-01T04:12:32.437 INFO:teuthology.orchestra.run.smithi153.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-11-01T04:12:32.438 INFO:teuthology.orchestra.run.smithi153.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-11-01T04:12:32.439 INFO:teuthology.orchestra.run.smithi181.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-11-01T04:12:32.440 INFO:teuthology.orchestra.run.smithi181.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-11-01T04:12:32.440 INFO:teuthology.orchestra.run.smithi181.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-11-01T04:12:32.441 INFO:teuthology.orchestra.run.smithi173.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-11-01T04:12:32.441 INFO:teuthology.orchestra.run.smithi153.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-11-01T04:12:32.442 INFO:teuthology.orchestra.run.smithi181.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-11-01T04:12:32.442 INFO:teuthology.orchestra.run.smithi181.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-11-01T04:12:32.442 INFO:teuthology.orchestra.run.smithi181.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-11-01T04:12:32.443 INFO:teuthology.orchestra.run.smithi173.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-11-01T04:12:32.443 INFO:teuthology.orchestra.run.smithi173.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-11-01T04:12:32.444 INFO:teuthology.orchestra.run.smithi181.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-11-01T04:12:32.444 INFO:teuthology.orchestra.run.smithi181.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-11-01T04:12:32.448 INFO:teuthology.orchestra.run.smithi181.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-11-01T04:12:32.458 INFO:teuthology.orchestra.run.smithi173.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-11-01T04:12:32.458 INFO:teuthology.orchestra.run.smithi173.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-11-01T04:12:32.461 INFO:teuthology.orchestra.run.smithi173.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6776-g420cd129-1jammy [3,130 kB] 2023-11-01T04:12:32.461 INFO:teuthology.orchestra.run.smithi181.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-11-01T04:12:32.463 INFO:teuthology.orchestra.run.smithi181.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-11-01T04:12:32.463 INFO:teuthology.orchestra.run.smithi181.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-11-01T04:12:32.465 INFO:teuthology.orchestra.run.smithi173.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-11-01T04:12:32.465 INFO:teuthology.orchestra.run.smithi173.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-11-01T04:12:32.465 INFO:teuthology.orchestra.run.smithi173.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-11-01T04:12:32.467 INFO:teuthology.orchestra.run.smithi181.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-11-01T04:12:32.467 INFO:teuthology.orchestra.run.smithi173.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-11-01T04:12:32.468 INFO:teuthology.orchestra.run.smithi173.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-11-01T04:12:32.469 INFO:teuthology.orchestra.run.smithi181.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-11-01T04:12:32.473 INFO:teuthology.orchestra.run.smithi181.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-11-01T04:12:32.475 INFO:teuthology.orchestra.run.smithi181.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-11-01T04:12:32.479 INFO:teuthology.orchestra.run.smithi181.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-11-01T04:12:32.480 INFO:teuthology.orchestra.run.smithi181.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-11-01T04:12:32.481 INFO:teuthology.orchestra.run.smithi181.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-11-01T04:12:32.492 INFO:teuthology.orchestra.run.smithi153.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-11-01T04:12:32.493 INFO:teuthology.orchestra.run.smithi153.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-11-01T04:12:32.494 INFO:teuthology.orchestra.run.smithi153.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-11-01T04:12:32.494 INFO:teuthology.orchestra.run.smithi153.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-11-01T04:12:32.495 INFO:teuthology.orchestra.run.smithi153.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-11-01T04:12:32.495 INFO:teuthology.orchestra.run.smithi153.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-11-01T04:12:32.496 INFO:teuthology.orchestra.run.smithi181.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-11-01T04:12:32.497 INFO:teuthology.orchestra.run.smithi181.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-11-01T04:12:32.499 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6776-g420cd129-1jammy [3,130 kB] 2023-11-01T04:12:32.503 INFO:teuthology.orchestra.run.smithi181.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-11-01T04:12:32.504 INFO:teuthology.orchestra.run.smithi181.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-11-01T04:12:32.504 INFO:teuthology.orchestra.run.smithi181.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-11-01T04:12:32.506 INFO:teuthology.orchestra.run.smithi181.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-11-01T04:12:32.506 INFO:teuthology.orchestra.run.smithi181.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-11-01T04:12:32.512 INFO:teuthology.orchestra.run.smithi153.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-11-01T04:12:32.514 INFO:teuthology.orchestra.run.smithi153.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-11-01T04:12:32.514 INFO:teuthology.orchestra.run.smithi153.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-11-01T04:12:32.529 INFO:teuthology.orchestra.run.smithi153.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-11-01T04:12:32.529 INFO:teuthology.orchestra.run.smithi153.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-11-01T04:12:32.529 INFO:teuthology.orchestra.run.smithi153.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-11-01T04:12:32.530 INFO:teuthology.orchestra.run.smithi153.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-11-01T04:12:32.530 INFO:teuthology.orchestra.run.smithi153.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-11-01T04:12:32.530 INFO:teuthology.orchestra.run.smithi153.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-11-01T04:12:32.544 INFO:teuthology.orchestra.run.smithi153.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-11-01T04:12:32.544 INFO:teuthology.orchestra.run.smithi153.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-11-01T04:12:32.547 INFO:teuthology.orchestra.run.smithi153.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-11-01T04:12:32.560 INFO:teuthology.orchestra.run.smithi153.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-11-01T04:12:32.562 INFO:teuthology.orchestra.run.smithi153.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-11-01T04:12:32.562 INFO:teuthology.orchestra.run.smithi153.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-11-01T04:12:32.565 INFO:teuthology.orchestra.run.smithi153.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-11-01T04:12:32.567 INFO:teuthology.orchestra.run.smithi153.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-11-01T04:12:32.571 INFO:teuthology.orchestra.run.smithi153.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-11-01T04:12:32.574 INFO:teuthology.orchestra.run.smithi153.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-11-01T04:12:32.577 INFO:teuthology.orchestra.run.smithi153.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-11-01T04:12:32.578 INFO:teuthology.orchestra.run.smithi153.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-11-01T04:12:32.578 INFO:teuthology.orchestra.run.smithi153.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-11-01T04:12:32.578 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-11-01T04:12:32.592 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-11-01T04:12:32.594 INFO:teuthology.orchestra.run.smithi153.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-11-01T04:12:32.594 INFO:teuthology.orchestra.run.smithi153.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-11-01T04:12:32.600 INFO:teuthology.orchestra.run.smithi153.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-11-01T04:12:32.601 INFO:teuthology.orchestra.run.smithi153.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-11-01T04:12:32.601 INFO:teuthology.orchestra.run.smithi153.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-11-01T04:12:32.603 INFO:teuthology.orchestra.run.smithi153.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-11-01T04:12:32.603 INFO:teuthology.orchestra.run.smithi153.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-11-01T04:12:32.619 INFO:teuthology.orchestra.run.smithi181.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6776-g420cd129-1jammy [3,130 kB] 2023-11-01T04:12:32.665 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-11-01T04:12:32.685 INFO:teuthology.orchestra.run.smithi173.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6776-g420cd129-1jammy [688 kB] 2023-11-01T04:12:32.753 INFO:teuthology.orchestra.run.smithi173.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6776-g420cd129-1jammy [356 kB] 2023-11-01T04:12:32.765 INFO:teuthology.orchestra.run.smithi173.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6776-g420cd129-1jammy [32.8 kB] 2023-11-01T04:12:32.804 INFO:teuthology.orchestra.run.smithi173.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6776-g420cd129-1jammy [180 kB] 2023-11-01T04:12:32.813 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6776-g420cd129-1jammy [688 kB] 2023-11-01T04:12:32.816 INFO:teuthology.orchestra.run.smithi173.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6776-g420cd129-1jammy [62.4 kB] 2023-11-01T04:12:32.818 INFO:teuthology.orchestra.run.smithi173.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6776-g420cd129-1jammy [330 kB] 2023-11-01T04:12:32.833 INFO:teuthology.orchestra.run.smithi173.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6776-g420cd129-1jammy [6,220 kB] 2023-11-01T04:12:32.859 INFO:teuthology.orchestra.run.smithi190.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-11-01T04:12:32.870 INFO:teuthology.orchestra.run.smithi190.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-11-01T04:12:32.897 INFO:teuthology.orchestra.run.smithi190.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6776-g420cd129-1jammy [356 kB] 2023-11-01T04:12:32.899 INFO:teuthology.orchestra.run.smithi190.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-11-01T04:12:32.907 INFO:teuthology.orchestra.run.smithi190.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-11-01T04:12:32.907 INFO:teuthology.orchestra.run.smithi190.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-11-01T04:12:32.909 INFO:teuthology.orchestra.run.smithi190.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-11-01T04:12:32.937 INFO:teuthology.orchestra.run.smithi181.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6776-g420cd129-1jammy [688 kB] 2023-11-01T04:12:32.987 INFO:teuthology.orchestra.run.smithi190.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6776-g420cd129-1jammy [32.8 kB] 2023-11-01T04:12:32.989 INFO:teuthology.orchestra.run.smithi190.stdout:Get:17 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6776-g420cd129-1jammy [180 kB] 2023-11-01T04:12:33.001 INFO:teuthology.orchestra.run.smithi190.stdout:Get:18 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6776-g420cd129-1jammy [62.4 kB] 2023-11-01T04:12:33.006 INFO:teuthology.orchestra.run.smithi190.stdout:Get:19 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6776-g420cd129-1jammy [330 kB] 2023-11-01T04:12:33.012 INFO:teuthology.orchestra.run.smithi181.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6776-g420cd129-1jammy [356 kB] 2023-11-01T04:12:33.023 INFO:teuthology.orchestra.run.smithi181.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6776-g420cd129-1jammy [32.8 kB] 2023-11-01T04:12:33.025 INFO:teuthology.orchestra.run.smithi181.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6776-g420cd129-1jammy [180 kB] 2023-11-01T04:12:33.072 INFO:teuthology.orchestra.run.smithi190.stdout:Get:20 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6776-g420cd129-1jammy [6,220 kB] 2023-11-01T04:12:33.079 INFO:teuthology.orchestra.run.smithi181.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6776-g420cd129-1jammy [62.4 kB] 2023-11-01T04:12:33.081 INFO:teuthology.orchestra.run.smithi181.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6776-g420cd129-1jammy [330 kB] 2023-11-01T04:12:33.090 INFO:teuthology.orchestra.run.smithi190.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-11-01T04:12:33.095 INFO:teuthology.orchestra.run.smithi181.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6776-g420cd129-1jammy [6,220 kB] 2023-11-01T04:12:33.302 INFO:teuthology.orchestra.run.smithi190.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-11-01T04:12:33.315 INFO:teuthology.orchestra.run.smithi190.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-11-01T04:12:33.395 INFO:teuthology.orchestra.run.smithi173.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6776-g420cd129-1jammy [112 kB] 2023-11-01T04:12:33.398 INFO:teuthology.orchestra.run.smithi173.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6776-g420cd129-1jammy [439 kB] 2023-11-01T04:12:33.433 INFO:teuthology.orchestra.run.smithi190.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-11-01T04:12:33.441 INFO:teuthology.orchestra.run.smithi190.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-11-01T04:12:33.449 INFO:teuthology.orchestra.run.smithi173.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6776-g420cd129-1jammy [22.6 MB] 2023-11-01T04:12:33.452 INFO:teuthology.orchestra.run.smithi190.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-11-01T04:12:33.469 INFO:teuthology.orchestra.run.smithi190.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-11-01T04:12:33.483 INFO:teuthology.orchestra.run.smithi190.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-11-01T04:12:33.487 INFO:teuthology.orchestra.run.smithi190.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-11-01T04:12:33.492 INFO:teuthology.orchestra.run.smithi190.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-11-01T04:12:33.594 INFO:teuthology.orchestra.run.smithi190.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-11-01T04:12:33.692 INFO:teuthology.orchestra.run.smithi181.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6776-g420cd129-1jammy [112 kB] 2023-11-01T04:12:33.694 INFO:teuthology.orchestra.run.smithi181.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6776-g420cd129-1jammy [439 kB] 2023-11-01T04:12:33.748 INFO:teuthology.orchestra.run.smithi181.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6776-g420cd129-1jammy [22.6 MB] 2023-11-01T04:12:33.769 INFO:teuthology.orchestra.run.smithi190.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-11-01T04:12:33.864 INFO:teuthology.orchestra.run.smithi190.stdout:Get:33 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6776-g420cd129-1jammy [112 kB] 2023-11-01T04:12:33.875 INFO:teuthology.orchestra.run.smithi190.stdout:Get:34 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6776-g420cd129-1jammy [439 kB] 2023-11-01T04:12:33.932 INFO:teuthology.orchestra.run.smithi190.stdout:Get:35 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6776-g420cd129-1jammy [22.6 MB] 2023-11-01T04:12:33.936 INFO:teuthology.orchestra.run.smithi190.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-11-01T04:12:33.957 INFO:teuthology.orchestra.run.smithi190.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-11-01T04:12:34.092 INFO:teuthology.orchestra.run.smithi190.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-11-01T04:12:34.108 INFO:teuthology.orchestra.run.smithi190.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-11-01T04:12:34.116 INFO:teuthology.orchestra.run.smithi190.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-11-01T04:12:34.189 INFO:teuthology.orchestra.run.smithi190.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-11-01T04:12:34.216 INFO:teuthology.orchestra.run.smithi190.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-11-01T04:12:34.224 INFO:teuthology.orchestra.run.smithi190.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-11-01T04:12:34.347 INFO:teuthology.orchestra.run.smithi190.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-11-01T04:12:34.348 INFO:teuthology.orchestra.run.smithi190.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-11-01T04:12:34.517 INFO:teuthology.orchestra.run.smithi190.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-11-01T04:12:34.706 INFO:teuthology.orchestra.run.smithi190.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-11-01T04:12:34.839 INFO:teuthology.orchestra.run.smithi153.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6776-g420cd129-1jammy [3,130 kB] 2023-11-01T04:12:34.860 INFO:teuthology.orchestra.run.smithi190.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-11-01T04:12:34.965 INFO:teuthology.orchestra.run.smithi190.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-11-01T04:12:34.987 INFO:teuthology.orchestra.run.smithi190.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-11-01T04:12:35.019 INFO:teuthology.orchestra.run.smithi190.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-11-01T04:12:35.032 INFO:teuthology.orchestra.run.smithi190.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-11-01T04:12:35.072 INFO:teuthology.orchestra.run.smithi190.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-11-01T04:12:35.081 INFO:teuthology.orchestra.run.smithi190.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-11-01T04:12:35.111 INFO:teuthology.orchestra.run.smithi190.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-11-01T04:12:35.339 INFO:teuthology.orchestra.run.smithi190.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-11-01T04:12:35.504 INFO:teuthology.orchestra.run.smithi190.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-11-01T04:12:35.669 INFO:teuthology.orchestra.run.smithi173.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6776-g420cd129-1jammy [6,853 kB] 2023-11-01T04:12:35.854 INFO:teuthology.orchestra.run.smithi190.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-11-01T04:12:35.871 INFO:teuthology.orchestra.run.smithi190.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-11-01T04:12:35.879 INFO:teuthology.orchestra.run.smithi190.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-11-01T04:12:35.889 INFO:teuthology.orchestra.run.smithi181.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6776-g420cd129-1jammy [6,853 kB] 2023-11-01T04:12:35.942 INFO:teuthology.orchestra.run.smithi190.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-11-01T04:12:36.239 INFO:teuthology.orchestra.run.smithi190.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-11-01T04:12:36.248 INFO:teuthology.orchestra.run.smithi190.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-11-01T04:12:36.249 INFO:teuthology.orchestra.run.smithi190.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-11-01T04:12:36.271 INFO:teuthology.orchestra.run.smithi173.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6776-g420cd129-1jammy [238 kB] 2023-11-01T04:12:36.282 INFO:teuthology.orchestra.run.smithi173.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6776-g420cd129-1jammy [128 kB] 2023-11-01T04:12:36.287 INFO:teuthology.orchestra.run.smithi173.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6776-g420cd129-1jammy [1,496 kB] 2023-11-01T04:12:36.315 INFO:teuthology.orchestra.run.smithi190.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-11-01T04:12:36.315 INFO:teuthology.orchestra.run.smithi190.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-11-01T04:12:36.317 INFO:teuthology.orchestra.run.smithi190.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-11-01T04:12:36.413 INFO:teuthology.orchestra.run.smithi181.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6776-g420cd129-1jammy [238 kB] 2023-11-01T04:12:36.437 INFO:teuthology.orchestra.run.smithi173.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6776-g420cd129-1jammy [9,863 kB] 2023-11-01T04:12:36.446 INFO:teuthology.orchestra.run.smithi181.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6776-g420cd129-1jammy [128 kB] 2023-11-01T04:12:36.451 INFO:teuthology.orchestra.run.smithi181.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6776-g420cd129-1jammy [1,496 kB] 2023-11-01T04:12:36.482 INFO:teuthology.orchestra.run.smithi190.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-11-01T04:12:36.595 INFO:teuthology.orchestra.run.smithi181.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6776-g420cd129-1jammy [9,863 kB] 2023-11-01T04:12:36.659 INFO:teuthology.orchestra.run.smithi153.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6776-g420cd129-1jammy [688 kB] 2023-11-01T04:12:36.672 INFO:teuthology.orchestra.run.smithi190.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-11-01T04:12:36.735 INFO:teuthology.orchestra.run.smithi190.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-11-01T04:12:36.770 INFO:teuthology.orchestra.run.smithi190.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-11-01T04:12:36.829 INFO:teuthology.orchestra.run.smithi190.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-11-01T04:12:37.011 INFO:teuthology.orchestra.run.smithi190.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-11-01T04:12:37.046 INFO:teuthology.orchestra.run.smithi190.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-11-01T04:12:37.047 INFO:teuthology.orchestra.run.smithi190.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6776-g420cd129-1jammy [6,853 kB] 2023-11-01T04:12:37.060 INFO:teuthology.orchestra.run.smithi190.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-11-01T04:12:37.063 INFO:teuthology.orchestra.run.smithi153.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6776-g420cd129-1jammy [356 kB] 2023-11-01T04:12:37.067 INFO:teuthology.orchestra.run.smithi190.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-11-01T04:12:37.108 INFO:teuthology.orchestra.run.smithi190.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-11-01T04:12:37.132 INFO:teuthology.orchestra.run.smithi190.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-11-01T04:12:37.297 INFO:teuthology.orchestra.run.smithi153.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6776-g420cd129-1jammy [32.8 kB] 2023-11-01T04:12:37.303 INFO:teuthology.orchestra.run.smithi153.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6776-g420cd129-1jammy [180 kB] 2023-11-01T04:12:37.330 INFO:teuthology.orchestra.run.smithi190.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-11-01T04:12:37.392 INFO:teuthology.orchestra.run.smithi153.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6776-g420cd129-1jammy [62.4 kB] 2023-11-01T04:12:37.457 INFO:teuthology.orchestra.run.smithi181.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6776-g420cd129-1jammy [31.7 MB] 2023-11-01T04:12:37.462 INFO:teuthology.orchestra.run.smithi153.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6776-g420cd129-1jammy [330 kB] 2023-11-01T04:12:37.468 INFO:teuthology.orchestra.run.smithi173.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6776-g420cd129-1jammy [31.7 MB] 2023-11-01T04:12:37.634 INFO:teuthology.orchestra.run.smithi153.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6776-g420cd129-1jammy [6,220 kB] 2023-11-01T04:12:37.812 INFO:teuthology.orchestra.run.smithi190.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-11-01T04:12:37.813 INFO:teuthology.orchestra.run.smithi190.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-11-01T04:12:37.823 INFO:teuthology.orchestra.run.smithi190.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-11-01T04:12:37.827 INFO:teuthology.orchestra.run.smithi190.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-11-01T04:12:37.857 INFO:teuthology.orchestra.run.smithi190.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-11-01T04:12:37.871 INFO:teuthology.orchestra.run.smithi190.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-11-01T04:12:37.892 INFO:teuthology.orchestra.run.smithi190.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6776-g420cd129-1jammy [238 kB] 2023-11-01T04:12:37.907 INFO:teuthology.orchestra.run.smithi190.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6776-g420cd129-1jammy [128 kB] 2023-11-01T04:12:37.926 INFO:teuthology.orchestra.run.smithi190.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6776-g420cd129-1jammy [1,496 kB] 2023-11-01T04:12:37.998 INFO:teuthology.orchestra.run.smithi190.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-11-01T04:12:38.000 INFO:teuthology.orchestra.run.smithi190.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-11-01T04:12:38.001 INFO:teuthology.orchestra.run.smithi190.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-11-01T04:12:38.019 INFO:teuthology.orchestra.run.smithi190.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-11-01T04:12:38.136 INFO:teuthology.orchestra.run.smithi190.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6776-g420cd129-1jammy [9,863 kB] 2023-11-01T04:12:38.187 INFO:teuthology.orchestra.run.smithi190.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-11-01T04:12:39.543 INFO:teuthology.orchestra.run.smithi190.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6776-g420cd129-1jammy [31.7 MB] 2023-11-01T04:12:40.307 INFO:teuthology.orchestra.run.smithi181.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6776-g420cd129-1jammy [14.1 kB] 2023-11-01T04:12:40.307 INFO:teuthology.orchestra.run.smithi181.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6776-g420cd129-1jammy [839 kB] 2023-11-01T04:12:40.324 INFO:teuthology.orchestra.run.smithi181.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6776-g420cd129-1jammy [2,146 kB] 2023-11-01T04:12:40.485 INFO:teuthology.orchestra.run.smithi181.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6776-g420cd129-1jammy [284 kB] 2023-11-01T04:12:40.495 INFO:teuthology.orchestra.run.smithi181.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6776-g420cd129-1jammy [138 kB] 2023-11-01T04:12:40.499 INFO:teuthology.orchestra.run.smithi181.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6776-g420cd129-1jammy [1,906 kB] 2023-11-01T04:12:40.662 INFO:teuthology.orchestra.run.smithi181.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6776-g420cd129-1jammy [8,625 kB] 2023-11-01T04:12:40.712 INFO:teuthology.orchestra.run.smithi173.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6776-g420cd129-1jammy [14.1 kB] 2023-11-01T04:12:40.787 INFO:teuthology.orchestra.run.smithi173.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6776-g420cd129-1jammy [839 kB] 2023-11-01T04:12:40.803 INFO:teuthology.orchestra.run.smithi173.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6776-g420cd129-1jammy [2,146 kB] 2023-11-01T04:12:40.944 INFO:teuthology.orchestra.run.smithi173.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6776-g420cd129-1jammy [284 kB] 2023-11-01T04:12:40.998 INFO:teuthology.orchestra.run.smithi173.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6776-g420cd129-1jammy [138 kB] 2023-11-01T04:12:41.010 INFO:teuthology.orchestra.run.smithi173.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6776-g420cd129-1jammy [1,906 kB] 2023-11-01T04:12:41.090 INFO:teuthology.orchestra.run.smithi153.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6776-g420cd129-1jammy [112 kB] 2023-11-01T04:12:41.154 INFO:teuthology.orchestra.run.smithi153.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6776-g420cd129-1jammy [439 kB] 2023-11-01T04:12:41.183 INFO:teuthology.orchestra.run.smithi173.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6776-g420cd129-1jammy [8,625 kB] 2023-11-01T04:12:41.384 INFO:teuthology.orchestra.run.smithi153.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6776-g420cd129-1jammy [22.6 MB] 2023-11-01T04:12:41.432 INFO:teuthology.orchestra.run.smithi181.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6776-g420cd129-1jammy [14.2 kB] 2023-11-01T04:12:41.432 INFO:teuthology.orchestra.run.smithi181.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6776-g420cd129-1jammy [48.2 MB] 2023-11-01T04:12:42.029 INFO:teuthology.orchestra.run.smithi173.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6776-g420cd129-1jammy [14.2 kB] 2023-11-01T04:12:42.030 INFO:teuthology.orchestra.run.smithi173.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6776-g420cd129-1jammy [48.2 MB] 2023-11-01T04:12:43.807 INFO:teuthology.orchestra.run.smithi190.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6776-g420cd129-1jammy [14.1 kB] 2023-11-01T04:12:43.807 INFO:teuthology.orchestra.run.smithi190.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6776-g420cd129-1jammy [839 kB] 2023-11-01T04:12:43.824 INFO:teuthology.orchestra.run.smithi190.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6776-g420cd129-1jammy [2,146 kB] 2023-11-01T04:12:44.030 INFO:teuthology.orchestra.run.smithi190.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6776-g420cd129-1jammy [284 kB] 2023-11-01T04:12:44.065 INFO:teuthology.orchestra.run.smithi190.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6776-g420cd129-1jammy [138 kB] 2023-11-01T04:12:44.073 INFO:teuthology.orchestra.run.smithi190.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6776-g420cd129-1jammy [1,906 kB] 2023-11-01T04:12:44.303 INFO:teuthology.orchestra.run.smithi190.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6776-g420cd129-1jammy [8,625 kB] 2023-11-01T04:12:45.229 INFO:teuthology.orchestra.run.smithi190.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6776-g420cd129-1jammy [14.2 kB] 2023-11-01T04:12:45.229 INFO:teuthology.orchestra.run.smithi190.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6776-g420cd129-1jammy [48.2 MB] 2023-11-01T04:12:46.697 INFO:teuthology.orchestra.run.smithi181.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6776-g420cd129-1jammy [120 kB] 2023-11-01T04:12:46.699 INFO:teuthology.orchestra.run.smithi181.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6776-g420cd129-1jammy [38.3 kB] 2023-11-01T04:12:46.700 INFO:teuthology.orchestra.run.smithi181.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6776-g420cd129-1jammy [12.3 MB] 2023-11-01T04:12:47.116 INFO:teuthology.orchestra.run.smithi153.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6776-g420cd129-1jammy [6,853 kB] 2023-11-01T04:12:47.858 INFO:teuthology.orchestra.run.smithi181.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6776-g420cd129-1jammy [91.6 kB] 2023-11-01T04:12:47.914 INFO:teuthology.orchestra.run.smithi153.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6776-g420cd129-1jammy [238 kB] 2023-11-01T04:12:47.919 INFO:teuthology.orchestra.run.smithi153.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6776-g420cd129-1jammy [128 kB] 2023-11-01T04:12:47.922 INFO:teuthology.orchestra.run.smithi153.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6776-g420cd129-1jammy [1,496 kB] 2023-11-01T04:12:48.019 INFO:teuthology.orchestra.run.smithi153.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6776-g420cd129-1jammy [9,863 kB] 2023-11-01T04:12:48.417 INFO:teuthology.orchestra.run.smithi181.stdout:Fetched 184 MB in 16s (11.6 MB/s) 2023-11-01T04:12:48.660 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libfmt8:amd64. 2023-11-01T04:12:48.761 INFO:teuthology.orchestra.run.smithi181.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165985 files and directories currently installed.) 2023-11-01T04:12:48.767 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-11-01T04:12:48.788 INFO:teuthology.orchestra.run.smithi153.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6776-g420cd129-1jammy [31.7 MB] 2023-11-01T04:12:48.851 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-11-01T04:12:49.179 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-11-01T04:12:49.205 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-11-01T04:12:49.246 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-01T04:12:49.549 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-11-01T04:12:49.564 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-11-01T04:12:49.599 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-01T04:12:49.707 INFO:teuthology.orchestra.run.smithi173.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6776-g420cd129-1jammy [120 kB] 2023-11-01T04:12:49.710 INFO:teuthology.orchestra.run.smithi173.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6776-g420cd129-1jammy [38.3 kB] 2023-11-01T04:12:49.711 INFO:teuthology.orchestra.run.smithi173.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6776-g420cd129-1jammy [12.3 MB] 2023-11-01T04:12:49.799 INFO:teuthology.orchestra.run.smithi190.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6776-g420cd129-1jammy [120 kB] 2023-11-01T04:12:49.873 INFO:teuthology.orchestra.run.smithi190.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6776-g420cd129-1jammy [38.3 kB] 2023-11-01T04:12:49.874 INFO:teuthology.orchestra.run.smithi190.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6776-g420cd129-1jammy [12.3 MB] 2023-11-01T04:12:49.952 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-11-01T04:12:49.978 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-11-01T04:12:50.019 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-01T04:12:50.464 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-11-01T04:12:50.490 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-01T04:12:50.539 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:12:50.825 INFO:teuthology.orchestra.run.smithi190.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6776-g420cd129-1jammy [91.6 kB] 2023-11-01T04:12:50.846 INFO:teuthology.orchestra.run.smithi153.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6776-g420cd129-1jammy [14.1 kB] 2023-11-01T04:12:50.847 INFO:teuthology.orchestra.run.smithi153.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6776-g420cd129-1jammy [839 kB] 2023-11-01T04:12:50.863 INFO:teuthology.orchestra.run.smithi153.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6776-g420cd129-1jammy [2,146 kB] 2023-11-01T04:12:50.900 INFO:teuthology.orchestra.run.smithi153.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6776-g420cd129-1jammy [284 kB] 2023-11-01T04:12:50.904 INFO:teuthology.orchestra.run.smithi153.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6776-g420cd129-1jammy [138 kB] 2023-11-01T04:12:50.906 INFO:teuthology.orchestra.run.smithi153.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6776-g420cd129-1jammy [1,906 kB] 2023-11-01T04:12:50.909 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-11-01T04:12:50.935 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-01T04:12:50.960 INFO:teuthology.orchestra.run.smithi153.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6776-g420cd129-1jammy [8,625 kB] 2023-11-01T04:12:50.975 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:12:51.313 INFO:teuthology.orchestra.run.smithi153.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6776-g420cd129-1jammy [14.2 kB] 2023-11-01T04:12:51.314 INFO:teuthology.orchestra.run.smithi153.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6776-g420cd129-1jammy [48.2 MB] 2023-11-01T04:12:51.345 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-11-01T04:12:51.371 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-01T04:12:51.383 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 184 MB in 19s (9,754 kB/s) 2023-11-01T04:12:51.411 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:12:51.517 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libfmt8:amd64. 2023-11-01T04:12:51.610 INFO:teuthology.orchestra.run.smithi190.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165985 files and directories currently installed.) 2023-11-01T04:12:51.615 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-11-01T04:12:51.699 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-11-01T04:12:51.790 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-11-01T04:12:51.816 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-11-01T04:12:51.856 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-01T04:12:52.002 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-11-01T04:12:52.028 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-11-01T04:12:52.060 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-01T04:12:52.143 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package librados2. 2023-11-01T04:12:52.169 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../008-librados2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:52.201 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking librados2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:52.414 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-11-01T04:12:52.440 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-11-01T04:12:52.488 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-01T04:12:52.637 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package librbd1. 2023-11-01T04:12:52.663 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../009-librbd1_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:52.703 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking librbd1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:52.801 INFO:teuthology.orchestra.run.smithi173.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6776-g420cd129-1jammy [91.6 kB] 2023-11-01T04:12:52.833 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-11-01T04:12:52.859 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-11-01T04:12:52.900 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-01T04:12:53.157 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libcephfs2. 2023-11-01T04:12:53.183 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:53.223 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libcephfs2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:53.295 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-11-01T04:12:53.318 INFO:teuthology.orchestra.run.smithi153.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6776-g420cd129-1jammy [120 kB] 2023-11-01T04:12:53.321 INFO:teuthology.orchestra.run.smithi153.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6776-g420cd129-1jammy [38.3 kB] 2023-11-01T04:12:53.321 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-01T04:12:53.322 INFO:teuthology.orchestra.run.smithi153.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6776-g420cd129-1jammy [12.3 MB] 2023-11-01T04:12:53.353 INFO:teuthology.orchestra.run.smithi173.stdout:Fetched 184 MB in 21s (8,838 kB/s) 2023-11-01T04:12:53.361 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:12:53.484 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-rados. 2023-11-01T04:12:53.492 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libfmt8:amd64. 2023-11-01T04:12:53.499 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:53.534 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-rados (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:53.591 INFO:teuthology.orchestra.run.smithi173.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165985 files and directories currently installed.) 2023-11-01T04:12:53.596 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-11-01T04:12:53.617 INFO:teuthology.orchestra.run.smithi153.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/DBG_maintain_prefix_itr_2/420cd129397301b1521ceb74e1bdbd9b855d2189/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6776-g420cd129-1jammy [91.6 kB] 2023-11-01T04:12:53.682 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-11-01T04:12:53.739 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-11-01T04:12:53.765 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-01T04:12:53.806 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:12:53.994 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-11-01T04:12:54.009 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-11-01T04:12:54.044 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-01T04:12:54.145 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-11-01T04:12:54.167 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 184 MB in 22s (8,559 kB/s) 2023-11-01T04:12:54.167 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-11-01T04:12:54.171 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:12:54.193 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-01T04:12:54.211 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-ceph-argparse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:54.234 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:12:54.302 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libfmt8:amd64. 2023-11-01T04:12:54.339 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-11-01T04:12:54.367 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-11-01T04:12:54.398 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-01T04:12:54.400 INFO:teuthology.orchestra.run.smithi153.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165985 files and directories currently installed.) 2023-11-01T04:12:54.405 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-11-01T04:12:54.464 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-cephfs. 2023-11-01T04:12:54.479 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:54.492 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-11-01T04:12:54.531 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-cephfs (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:54.603 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-11-01T04:12:54.630 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-11-01T04:12:54.670 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-01T04:12:54.726 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-11-01T04:12:54.753 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-11-01T04:12:54.792 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-01T04:12:54.828 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-11-01T04:12:54.842 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-ceph-common. 2023-11-01T04:12:54.854 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-11-01T04:12:54.868 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:12:54.895 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-01T04:12:54.933 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:54.940 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librados2. 2023-11-01T04:12:54.966 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../008-librados2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:54.998 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librados2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:55.213 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-11-01T04:12:55.240 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-01T04:12:55.245 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-wcwidth. 2023-11-01T04:12:55.260 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-11-01T04:12:55.282 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:12:55.295 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-01T04:12:55.323 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-11-01T04:12:55.350 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-11-01T04:12:55.398 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-01T04:12:55.459 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librbd1. 2023-11-01T04:12:55.485 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../009-librbd1_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:55.525 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librbd1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:55.589 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-prettytable. 2023-11-01T04:12:55.605 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-11-01T04:12:55.639 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-11-01T04:12:55.685 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-11-01T04:12:55.711 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-01T04:12:55.751 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:12:55.768 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-11-01T04:12:55.794 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-11-01T04:12:55.834 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-01T04:12:55.875 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-rbd. 2023-11-01T04:12:55.888 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:55.925 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-rbd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:55.970 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libcephfs2. 2023-11-01T04:12:55.996 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:56.028 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libcephfs2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:56.104 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-11-01T04:12:56.130 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-01T04:12:56.171 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:12:56.270 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-11-01T04:12:56.280 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-11-01T04:12:56.296 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-11-01T04:12:56.306 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-01T04:12:56.336 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-01T04:12:56.362 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:12:56.451 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rados. 2023-11-01T04:12:56.477 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:56.507 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-11-01T04:12:56.530 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rados (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:56.534 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-11-01T04:12:56.566 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-01T04:12:56.664 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package librgw2. 2023-11-01T04:12:56.690 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../019-librgw2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:56.731 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking librgw2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:56.850 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-11-01T04:12:56.866 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-11-01T04:12:56.877 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:12:56.893 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-01T04:12:56.893 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package librados2. 2023-11-01T04:12:56.916 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-ceph-argparse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:56.919 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../008-librados2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:56.941 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:12:56.960 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking librados2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:57.168 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cephfs. 2023-11-01T04:12:57.184 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:57.227 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cephfs (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:57.250 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-rgw. 2023-11-01T04:12:57.276 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:57.308 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-rgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:57.319 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-11-01T04:12:57.345 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-11-01T04:12:57.386 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:12:57.404 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package librbd1. 2023-11-01T04:12:57.431 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../009-librbd1_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:57.471 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking librbd1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:57.522 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-ceph-common. 2023-11-01T04:12:57.537 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:12:57.571 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:57.611 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package liboath0:amd64. 2023-11-01T04:12:57.637 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-11-01T04:12:57.677 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-11-01T04:12:57.739 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-11-01T04:12:57.765 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-11-01T04:12:57.814 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-01T04:12:57.849 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-wcwidth. 2023-11-01T04:12:57.876 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-11-01T04:12:57.907 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-01T04:12:57.965 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libcephfs2. 2023-11-01T04:12:57.972 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libradosstriper1. 2023-11-01T04:12:57.992 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:57.998 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:58.040 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libcephfs2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:58.055 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libradosstriper1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:58.225 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librados2. 2023-11-01T04:12:58.227 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-prettytable. 2023-11-01T04:12:58.251 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../008-librados2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:58.253 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-11-01T04:12:58.293 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-11-01T04:12:58.300 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librados2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:58.507 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-common. 2023-11-01T04:12:58.533 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:58.562 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rbd. 2023-11-01T04:12:58.575 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:58.578 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:58.621 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rbd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:58.688 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-rados. 2023-11-01T04:12:58.688 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:58.718 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-rados (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:58.811 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librbd1. 2023-11-01T04:12:58.837 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../009-librbd1_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:58.886 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librbd1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:58.974 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-11-01T04:12:59.001 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-11-01T04:12:59.040 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-01T04:12:59.062 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-11-01T04:12:59.089 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:12:59.129 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-ceph-argparse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:59.326 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librgw2. 2023-11-01T04:12:59.341 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../019-librgw2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:59.381 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-cephfs. 2023-11-01T04:12:59.401 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librgw2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:59.408 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:59.448 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-cephfs (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:59.506 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libcephfs2. 2023-11-01T04:12:59.532 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:59.572 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libcephfs2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:59.851 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-ceph-common. 2023-11-01T04:12:59.856 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-base. 2023-11-01T04:12:59.877 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:12:59.882 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:59.883 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rados. 2023-11-01T04:12:59.909 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:59.912 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rgw. 2023-11-01T04:12:59.925 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:59.932 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-base (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:59.938 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:12:59.958 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rados (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:12:59.978 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:00.312 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-wcwidth. 2023-11-01T04:13:00.323 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package liboath0:amd64. 2023-11-01T04:13:00.339 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-11-01T04:13:00.350 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-11-01T04:13:00.369 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-11-01T04:13:00.387 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-01T04:13:00.389 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-11-01T04:13:00.395 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:00.444 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-ceph-argparse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:00.644 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-11-01T04:13:00.670 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-11-01T04:13:00.684 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libradosstriper1. 2023-11-01T04:13:00.699 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:00.706 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-prettytable. 2023-11-01T04:13:00.710 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-11-01T04:13:00.713 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cephfs. 2023-11-01T04:13:00.728 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:00.732 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-11-01T04:13:00.733 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libradosstriper1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:00.771 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cephfs (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:00.773 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-11-01T04:13:00.963 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-cheroot. 2023-11-01T04:13:00.989 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-11-01T04:13:01.021 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-01T04:13:01.034 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-rbd. 2023-11-01T04:13:01.049 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:01.091 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-ceph-common. 2023-11-01T04:13:01.092 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-rbd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:01.117 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:01.129 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-common. 2023-11-01T04:13:01.155 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:01.158 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:01.195 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:01.341 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-11-01T04:13:01.367 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-11-01T04:13:01.407 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-11-01T04:13:01.486 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-wcwidth. 2023-11-01T04:13:01.495 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-11-01T04:13:01.512 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-11-01T04:13:01.521 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-11-01T04:13:01.552 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-01T04:13:01.570 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-01T04:13:01.710 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-jaraco.text. 2023-11-01T04:13:01.737 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-11-01T04:13:01.777 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-11-01T04:13:01.880 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-prettytable. 2023-11-01T04:13:01.906 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-11-01T04:13:01.946 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-11-01T04:13:01.964 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package librgw2. 2023-11-01T04:13:01.990 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../019-librgw2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:02.039 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking librgw2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:02.105 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-11-01T04:13:02.131 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-11-01T04:13:02.171 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-11-01T04:13:02.183 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rbd. 2023-11-01T04:13:02.198 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:02.241 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rbd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:02.441 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-tempora. 2023-11-01T04:13:02.467 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-11-01T04:13:02.499 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-11-01T04:13:02.534 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-base. 2023-11-01T04:13:02.550 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-rgw. 2023-11-01T04:13:02.561 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:02.566 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:02.602 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-11-01T04:13:02.609 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-base (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:02.625 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-rgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:02.629 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-11-01T04:13:02.694 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-01T04:13:02.811 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-portend. 2023-11-01T04:13:02.837 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-11-01T04:13:02.877 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-11-01T04:13:02.961 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package liboath0:amd64. 2023-11-01T04:13:02.987 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-11-01T04:13:03.028 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-11-01T04:13:03.030 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librgw2. 2023-11-01T04:13:03.056 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../019-librgw2_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:03.096 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librgw2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:03.180 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-11-01T04:13:03.206 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-11-01T04:13:03.246 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-11-01T04:13:03.338 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-11-01T04:13:03.365 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-11-01T04:13:03.405 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-11-01T04:13:03.524 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-cherrypy3. 2023-11-01T04:13:03.539 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libradosstriper1. 2023-11-01T04:13:03.540 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-11-01T04:13:03.565 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:03.574 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-11-01T04:13:03.614 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libradosstriper1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:03.641 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rgw. 2023-11-01T04:13:03.656 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:03.691 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cheroot. 2023-11-01T04:13:03.691 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:03.717 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-11-01T04:13:03.749 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-01T04:13:03.870 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-natsort. 2023-11-01T04:13:03.896 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-11-01T04:13:03.928 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-11-01T04:13:03.960 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package liboath0:amd64. 2023-11-01T04:13:03.986 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-11-01T04:13:04.018 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-11-01T04:13:04.019 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-11-01T04:13:04.033 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-11-01T04:13:04.068 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-11-01T04:13:04.076 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-common. 2023-11-01T04:13:04.102 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:04.142 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:04.338 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libradosstriper1. 2023-11-01T04:13:04.364 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:04.371 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.text. 2023-11-01T04:13:04.373 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-logutils. 2023-11-01T04:13:04.397 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-11-01T04:13:04.399 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-11-01T04:13:04.404 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libradosstriper1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:04.438 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-11-01T04:13:04.439 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-11-01T04:13:04.742 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-markupsafe. 2023-11-01T04:13:04.769 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-11-01T04:13:04.808 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-11-01T04:13:05.112 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-mako. 2023-11-01T04:13:05.138 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-11-01T04:13:05.178 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-01T04:13:05.457 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-11-01T04:13:05.483 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-11-01T04:13:05.524 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-11-01T04:13:05.575 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-common. 2023-11-01T04:13:05.602 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:05.641 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:05.793 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-tempora. 2023-11-01T04:13:05.809 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-11-01T04:13:05.843 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-11-01T04:13:06.064 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-base. 2023-11-01T04:13:06.091 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:06.121 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-portend. 2023-11-01T04:13:06.147 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-11-01T04:13:06.148 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-base (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:06.188 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-11-01T04:13:06.190 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-simplegeneric. 2023-11-01T04:13:06.218 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-11-01T04:13:06.257 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-11-01T04:13:06.490 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-11-01T04:13:06.516 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-11-01T04:13:06.559 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-singledispatch. 2023-11-01T04:13:06.573 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-11-01T04:13:06.586 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-11-01T04:13:06.626 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-11-01T04:13:06.876 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cherrypy3. 2023-11-01T04:13:06.902 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-11-01T04:13:06.912 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-webob. 2023-11-01T04:13:06.928 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-11-01T04:13:06.943 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-11-01T04:13:06.956 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-base. 2023-11-01T04:13:06.962 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-11-01T04:13:06.969 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-11-01T04:13:06.982 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:06.995 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-11-01T04:13:07.031 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-base (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:07.043 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-11-01T04:13:07.282 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-waitress. 2023-11-01T04:13:07.308 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-11-01T04:13:07.321 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-cheroot. 2023-11-01T04:13:07.337 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-11-01T04:13:07.352 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-01T04:13:07.371 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-01T04:13:07.488 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-natsort. 2023-11-01T04:13:07.514 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-11-01T04:13:07.554 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-11-01T04:13:07.626 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-tempita. 2023-11-01T04:13:07.653 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-11-01T04:13:07.666 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-11-01T04:13:07.692 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-11-01T04:13:07.693 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-11-01T04:13:07.701 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-11-01T04:13:07.717 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-11-01T04:13:07.724 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-11-01T04:13:07.768 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-11-01T04:13:07.866 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-logutils. 2023-11-01T04:13:07.892 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-11-01T04:13:07.932 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-11-01T04:13:07.996 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-paste. 2023-11-01T04:13:08.018 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-jaraco.text. 2023-11-01T04:13:08.022 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-11-01T04:13:08.038 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cheroot. 2023-11-01T04:13:08.045 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-11-01T04:13:08.062 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-11-01T04:13:08.064 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-11-01T04:13:08.085 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-11-01T04:13:08.104 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-01T04:13:08.202 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-markupsafe. 2023-11-01T04:13:08.217 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-11-01T04:13:08.251 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-11-01T04:13:08.366 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-11-01T04:13:08.388 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-11-01T04:13:08.392 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-11-01T04:13:08.415 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-11-01T04:13:08.416 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-11-01T04:13:08.432 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-11-01T04:13:08.442 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-11-01T04:13:08.454 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-11-01T04:13:08.482 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-11-01T04:13:08.529 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-mako. 2023-11-01T04:13:08.556 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-11-01T04:13:08.588 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-01T04:13:08.718 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pastedeploy. 2023-11-01T04:13:08.744 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-11-01T04:13:08.785 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-11-01T04:13:08.785 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.text. 2023-11-01T04:13:08.811 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-11-01T04:13:08.851 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-11-01T04:13:08.874 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-simplegeneric. 2023-11-01T04:13:08.889 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-11-01T04:13:08.915 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-tempora. 2023-11-01T04:13:08.932 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-11-01T04:13:08.942 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-11-01T04:13:08.982 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-11-01T04:13:09.088 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-webtest. 2023-11-01T04:13:09.114 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-11-01T04:13:09.121 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-11-01T04:13:09.147 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-11-01T04:13:09.154 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-11-01T04:13:09.187 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-11-01T04:13:09.218 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-singledispatch. 2023-11-01T04:13:09.226 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-portend. 2023-11-01T04:13:09.244 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-11-01T04:13:09.253 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-11-01T04:13:09.285 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-11-01T04:13:09.285 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-11-01T04:13:09.449 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pecan. 2023-11-01T04:13:09.474 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-tempora. 2023-11-01T04:13:09.476 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-11-01T04:13:09.500 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-11-01T04:13:09.532 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-11-01T04:13:09.587 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-webob. 2023-11-01T04:13:09.588 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-11-01T04:13:09.614 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-11-01T04:13:09.614 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-11-01T04:13:09.654 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-11-01T04:13:09.662 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-11-01T04:13:09.665 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-11-01T04:13:09.818 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-portend. 2023-11-01T04:13:09.846 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-11-01T04:13:09.995 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-certifi. 2023-11-01T04:13:10.022 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-11-01T04:13:10.023 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-waitress. 2023-11-01T04:13:10.027 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-11-01T04:13:10.032 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-cherrypy3. 2023-11-01T04:13:10.050 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-11-01T04:13:10.053 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-11-01T04:13:10.059 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-11-01T04:13:10.107 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-11-01T04:13:10.116 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-01T04:13:10.356 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-idna. 2023-11-01T04:13:10.379 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-11-01T04:13:10.382 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-11-01T04:13:10.406 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-11-01T04:13:10.422 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-idna (3.3-1) ... 2023-11-01T04:13:10.435 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-natsort. 2023-11-01T04:13:10.446 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-11-01T04:13:10.462 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-11-01T04:13:10.468 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-tempita. 2023-11-01T04:13:10.495 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-11-01T04:13:10.502 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-11-01T04:13:10.534 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-11-01T04:13:10.734 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-urllib3. 2023-11-01T04:13:10.760 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-11-01T04:13:10.766 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cherrypy3. 2023-11-01T04:13:10.792 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-11-01T04:13:10.796 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-paste. 2023-11-01T04:13:10.800 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-11-01T04:13:10.813 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-logutils. 2023-11-01T04:13:10.822 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-11-01T04:13:10.840 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-11-01T04:13:10.841 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-11-01T04:13:10.862 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-11-01T04:13:10.880 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-11-01T04:13:11.129 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-requests. 2023-11-01T04:13:11.156 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-11-01T04:13:11.178 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-natsort. 2023-11-01T04:13:11.190 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-11-01T04:13:11.195 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-01T04:13:11.204 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-11-01T04:13:11.217 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-11-01T04:13:11.236 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-11-01T04:13:11.258 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-markupsafe. 2023-11-01T04:13:11.282 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-11-01T04:13:11.284 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-11-01T04:13:11.333 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-11-01T04:13:11.506 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-werkzeug. 2023-11-01T04:13:11.522 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-logutils. 2023-11-01T04:13:11.533 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-11-01T04:13:11.548 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-11-01T04:13:11.573 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-01T04:13:11.584 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pastedeploy. 2023-11-01T04:13:11.597 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-11-01T04:13:11.611 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-11-01T04:13:11.651 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-11-01T04:13:11.660 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-mako. 2023-11-01T04:13:11.687 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-11-01T04:13:11.727 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-01T04:13:11.869 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-11-01T04:13:11.895 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:11.900 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-markupsafe. 2023-11-01T04:13:11.927 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-11-01T04:13:11.934 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:11.954 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-webtest. 2023-11-01T04:13:11.967 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-11-01T04:13:11.981 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-11-01T04:13:12.020 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-11-01T04:13:12.038 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-simplegeneric. 2023-11-01T04:13:12.065 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-11-01T04:13:12.097 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-11-01T04:13:12.256 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-11-01T04:13:12.269 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-mako. 2023-11-01T04:13:12.272 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:12.296 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-11-01T04:13:12.298 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pecan. 2023-11-01T04:13:12.314 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:12.314 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-11-01T04:13:12.336 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-01T04:13:12.348 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-11-01T04:13:12.375 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-singledispatch. 2023-11-01T04:13:12.401 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-11-01T04:13:12.441 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-11-01T04:13:12.631 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-simplegeneric. 2023-11-01T04:13:12.634 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-openssl. 2023-11-01T04:13:12.657 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-11-01T04:13:12.661 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-11-01T04:13:12.661 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-certifi. 2023-11-01T04:13:12.688 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-11-01T04:13:12.700 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-11-01T04:13:12.706 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-11-01T04:13:12.719 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-11-01T04:13:12.744 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-webob. 2023-11-01T04:13:12.770 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-11-01T04:13:12.810 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-11-01T04:13:12.984 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-singledispatch. 2023-11-01T04:13:12.999 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-11-01T04:13:13.042 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-11-01T04:13:13.053 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-mgr. 2023-11-01T04:13:13.056 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-idna. 2023-11-01T04:13:13.080 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:13.082 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-11-01T04:13:13.122 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-idna (3.3-1) ... 2023-11-01T04:13:13.136 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-mgr (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:13.205 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-waitress. 2023-11-01T04:13:13.231 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-11-01T04:13:13.274 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-01T04:13:13.336 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-webob. 2023-11-01T04:13:13.363 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-11-01T04:13:13.403 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-11-01T04:13:13.441 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-urllib3. 2023-11-01T04:13:13.468 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-11-01T04:13:13.506 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-mon. 2023-11-01T04:13:13.508 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-11-01T04:13:13.533 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:13.564 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-mon (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:13.566 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-tempita. 2023-11-01T04:13:13.592 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-11-01T04:13:13.633 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-11-01T04:13:13.772 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-waitress. 2023-11-01T04:13:13.799 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-11-01T04:13:13.843 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-01T04:13:13.861 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-requests. 2023-11-01T04:13:13.888 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-11-01T04:13:13.927 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-01T04:13:14.061 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-paste. 2023-11-01T04:13:14.087 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-11-01T04:13:14.127 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-11-01T04:13:14.134 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-tempita. 2023-11-01T04:13:14.160 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-11-01T04:13:14.200 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-11-01T04:13:14.222 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-werkzeug. 2023-11-01T04:13:14.249 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-11-01T04:13:14.288 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-01T04:13:14.463 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-11-01T04:13:14.468 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-osd. 2023-11-01T04:13:14.478 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-11-01T04:13:14.495 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:14.512 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-paste. 2023-11-01T04:13:14.522 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-11-01T04:13:14.534 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-osd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:14.538 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-11-01T04:13:14.578 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-11-01T04:13:14.583 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-11-01T04:13:14.599 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:14.641 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:14.825 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pastedeploy. 2023-11-01T04:13:14.851 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-11-01T04:13:14.891 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-11-01T04:13:14.932 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-11-01T04:13:14.958 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-11-01T04:13:14.996 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-11-01T04:13:15.012 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:15.046 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:15.081 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-11-01T04:13:15.194 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-webtest. 2023-11-01T04:13:15.220 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-11-01T04:13:15.261 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-11-01T04:13:15.374 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-openssl. 2023-11-01T04:13:15.384 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pastedeploy. 2023-11-01T04:13:15.401 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-11-01T04:13:15.410 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-11-01T04:13:15.440 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-11-01T04:13:15.451 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-11-01T04:13:15.555 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pecan. 2023-11-01T04:13:15.581 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-11-01T04:13:15.613 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-11-01T04:13:15.770 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-webtest. 2023-11-01T04:13:15.785 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr. 2023-11-01T04:13:15.797 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-11-01T04:13:15.811 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:15.837 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-11-01T04:13:15.851 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:15.968 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-certifi. 2023-11-01T04:13:15.994 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-11-01T04:13:16.026 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-11-01T04:13:16.157 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pecan. 2023-11-01T04:13:16.183 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-11-01T04:13:16.223 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-11-01T04:13:16.254 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mon. 2023-11-01T04:13:16.281 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:16.320 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mon (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:16.337 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-idna. 2023-11-01T04:13:16.364 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-11-01T04:13:16.387 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph. 2023-11-01T04:13:16.403 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../059-ceph_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:16.423 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-idna (3.3-1) ... 2023-11-01T04:13:16.446 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:16.569 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-certifi. 2023-11-01T04:13:16.596 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-11-01T04:13:16.636 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-11-01T04:13:16.757 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-urllib3. 2023-11-01T04:13:16.783 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-11-01T04:13:16.815 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-fuse. 2023-11-01T04:13:16.823 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-11-01T04:13:16.842 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:16.882 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:16.964 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-idna. 2023-11-01T04:13:16.979 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-11-01T04:13:17.014 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-idna (3.3-1) ... 2023-11-01T04:13:17.152 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-requests. 2023-11-01T04:13:17.178 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-11-01T04:13:17.218 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-01T04:13:17.224 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-osd. 2023-11-01T04:13:17.251 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:17.252 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-mds. 2023-11-01T04:13:17.278 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:17.316 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-osd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:17.318 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-mds (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:17.325 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-urllib3. 2023-11-01T04:13:17.352 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-11-01T04:13:17.392 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-11-01T04:13:17.529 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-werkzeug. 2023-11-01T04:13:17.555 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-11-01T04:13:17.596 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-01T04:13:17.663 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package cephadm. 2023-11-01T04:13:17.670 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-requests. 2023-11-01T04:13:17.685 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-11-01T04:13:17.689 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../062-cephadm_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:17.720 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-01T04:13:17.721 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:17.907 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-11-01T04:13:17.922 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:17.965 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:18.006 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-werkzeug. 2023-11-01T04:13:18.033 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-11-01T04:13:18.041 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-asyncssh. 2023-11-01T04:13:18.067 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-11-01T04:13:18.072 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-01T04:13:18.107 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-11-01T04:13:18.342 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-11-01T04:13:18.369 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:18.370 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-11-01T04:13:18.397 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:18.409 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:18.410 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python-babel-localedata. 2023-11-01T04:13:18.436 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:18.437 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-11-01T04:13:18.477 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-01T04:13:18.748 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-openssl. 2023-11-01T04:13:18.774 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-11-01T04:13:18.805 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-11-01T04:13:18.814 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-11-01T04:13:18.832 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:18.872 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:19.160 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-openssl. 2023-11-01T04:13:19.176 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-11-01T04:13:19.209 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr. 2023-11-01T04:13:19.210 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-11-01T04:13:19.236 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:19.243 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph. 2023-11-01T04:13:19.259 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../059-ceph_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:19.284 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:19.302 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:19.482 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-babel. 2023-11-01T04:13:19.510 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-11-01T04:13:19.538 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr. 2023-11-01T04:13:19.557 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-11-01T04:13:19.564 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:19.604 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:19.630 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-fuse. 2023-11-01T04:13:19.657 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:19.696 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:19.770 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mon. 2023-11-01T04:13:19.797 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:19.845 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mon (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:19.893 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-jinja2. 2023-11-01T04:13:19.921 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-11-01T04:13:19.960 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-11-01T04:13:20.008 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mon. 2023-11-01T04:13:20.024 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mds. 2023-11-01T04:13:20.034 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:20.040 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:20.074 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mds (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:20.089 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mon (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:20.405 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-11-01T04:13:20.432 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:20.452 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package cephadm. 2023-11-01T04:13:20.471 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:20.480 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../062-cephadm_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:20.518 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:20.801 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-bcrypt. 2023-11-01T04:13:20.815 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-osd. 2023-11-01T04:13:20.827 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-11-01T04:13:20.842 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:20.863 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-asyncssh. 2023-11-01T04:13:20.867 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-11-01T04:13:20.881 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-osd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:20.890 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-11-01T04:13:20.938 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-11-01T04:13:21.103 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-osd. 2023-11-01T04:13:21.111 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-repoze.lru. 2023-11-01T04:13:21.129 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:21.138 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-11-01T04:13:21.177 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-osd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:21.178 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-11-01T04:13:21.274 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-babel-localedata. 2023-11-01T04:13:21.301 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-11-01T04:13:21.341 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-01T04:13:21.489 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-routes. 2023-11-01T04:13:21.516 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-11-01T04:13:21.555 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-11-01T04:13:21.909 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-11-01T04:13:21.936 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:21.976 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:22.379 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-11-01T04:13:22.395 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-11-01T04:13:22.404 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-babel. 2023-11-01T04:13:22.429 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-01T04:13:22.432 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-11-01T04:13:22.470 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-11-01T04:13:22.757 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jinja2. 2023-11-01T04:13:22.784 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-11-01T04:13:22.793 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph. 2023-11-01T04:13:22.809 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../059-ceph_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:22.823 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-11-01T04:13:22.851 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:22.924 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-joblib. 2023-11-01T04:13:22.951 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-11-01T04:13:22.991 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-11-01T04:13:23.105 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph. 2023-11-01T04:13:23.132 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../059-ceph_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:23.163 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-fuse. 2023-11-01T04:13:23.176 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-11-01T04:13:23.180 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:23.189 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:23.203 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:23.229 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:23.243 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:23.336 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-11-01T04:13:23.363 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-11-01T04:13:23.402 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-11-01T04:13:23.549 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-fuse. 2023-11-01T04:13:23.574 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mds. 2023-11-01T04:13:23.576 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:23.596 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-bcrypt. 2023-11-01T04:13:23.601 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:23.616 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:23.623 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-11-01T04:13:23.649 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mds (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:23.663 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-11-01T04:13:23.680 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-sklearn. 2023-11-01T04:13:23.708 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-11-01T04:13:23.738 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-01T04:13:23.966 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-repoze.lru. 2023-11-01T04:13:23.993 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-11-01T04:13:24.027 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mds. 2023-11-01T04:13:24.032 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-11-01T04:13:24.043 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package cephadm. 2023-11-01T04:13:24.054 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:24.070 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../062-cephadm_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:24.102 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mds (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:24.240 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:24.335 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-routes. 2023-11-01T04:13:24.362 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-11-01T04:13:24.401 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-11-01T04:13:24.522 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-11-01T04:13:24.539 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package cephadm. 2023-11-01T04:13:24.549 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:24.565 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../062-cephadm_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:24.588 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:24.613 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-asyncssh. 2023-11-01T04:13:24.613 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:24.640 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-11-01T04:13:24.679 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-11-01T04:13:24.780 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-11-01T04:13:24.807 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:24.847 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:24.975 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-asyncssh. 2023-11-01T04:13:25.001 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-11-01T04:13:25.024 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python-babel-localedata. 2023-11-01T04:13:25.050 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-11-01T04:13:25.051 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-11-01T04:13:25.099 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-01T04:13:25.267 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-11-01T04:13:25.293 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-11-01T04:13:25.333 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-01T04:13:25.544 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-cachetools. 2023-11-01T04:13:25.570 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-11-01T04:13:25.610 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-11-01T04:13:25.653 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python-babel-localedata. 2023-11-01T04:13:25.679 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-11-01T04:13:25.735 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-01T04:13:25.978 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-joblib. 2023-11-01T04:13:25.996 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pyasn1. 2023-11-01T04:13:26.006 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-11-01T04:13:26.023 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-11-01T04:13:26.044 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-11-01T04:13:26.062 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-11-01T04:13:26.204 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-babel. 2023-11-01T04:13:26.231 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-11-01T04:13:26.270 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-11-01T04:13:26.373 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-11-01T04:13:26.389 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-11-01T04:13:26.408 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-11-01T04:13:26.423 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-11-01T04:13:26.434 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-11-01T04:13:26.474 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-11-01T04:13:26.615 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-jinja2. 2023-11-01T04:13:26.642 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-11-01T04:13:26.682 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-11-01T04:13:26.709 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-sklearn. 2023-11-01T04:13:26.737 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-11-01T04:13:26.774 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-babel. 2023-11-01T04:13:26.776 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-01T04:13:26.801 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-11-01T04:13:26.802 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-rsa. 2023-11-01T04:13:26.829 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-11-01T04:13:26.840 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-11-01T04:13:26.868 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-rsa (4.8-1) ... 2023-11-01T04:13:26.993 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-11-01T04:13:27.009 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:27.043 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:27.177 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jinja2. 2023-11-01T04:13:27.189 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-google-auth. 2023-11-01T04:13:27.204 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-11-01T04:13:27.216 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-11-01T04:13:27.243 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-11-01T04:13:27.255 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-11-01T04:13:27.355 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-bcrypt. 2023-11-01T04:13:27.382 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-11-01T04:13:27.422 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-11-01T04:13:27.442 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-11-01T04:13:27.456 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:27.492 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:27.575 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-11-01T04:13:27.588 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-11-01T04:13:27.602 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-11-01T04:13:27.615 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:27.641 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-01T04:13:27.655 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:27.733 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-repoze.lru. 2023-11-01T04:13:27.760 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-11-01T04:13:27.799 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-11-01T04:13:28.000 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-bcrypt. 2023-11-01T04:13:28.026 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-11-01T04:13:28.046 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-websocket. 2023-11-01T04:13:28.066 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-11-01T04:13:28.073 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-11-01T04:13:28.102 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-routes. 2023-11-01T04:13:28.112 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-11-01T04:13:28.129 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-11-01T04:13:28.169 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-11-01T04:13:28.344 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-repoze.lru. 2023-11-01T04:13:28.371 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-11-01T04:13:28.411 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-11-01T04:13:28.432 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-kubernetes. 2023-11-01T04:13:28.459 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-11-01T04:13:28.499 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-01T04:13:28.514 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-11-01T04:13:28.541 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:28.580 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:28.603 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cachetools. 2023-11-01T04:13:28.630 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-11-01T04:13:28.670 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-11-01T04:13:28.688 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-routes. 2023-11-01T04:13:28.715 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-11-01T04:13:28.755 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-11-01T04:13:28.950 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyasn1. 2023-11-01T04:13:28.977 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-11-01T04:13:28.984 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-11-01T04:13:29.008 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-11-01T04:13:29.011 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-11-01T04:13:29.050 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-11-01T04:13:29.050 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-01T04:13:29.065 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:29.088 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-11-01T04:13:29.104 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:29.108 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:29.137 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:29.295 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-11-01T04:13:29.322 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-11-01T04:13:29.352 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-11-01T04:13:29.434 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libonig5:amd64. 2023-11-01T04:13:29.450 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-11-01T04:13:29.490 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-01T04:13:29.536 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-11-01T04:13:29.552 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-11-01T04:13:29.562 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-joblib. 2023-11-01T04:13:29.589 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-11-01T04:13:29.595 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-01T04:13:29.629 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-11-01T04:13:29.664 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rsa. 2023-11-01T04:13:29.691 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-11-01T04:13:29.730 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rsa (4.8-1) ... 2023-11-01T04:13:29.835 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libjq1:amd64. 2023-11-01T04:13:29.862 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-11-01T04:13:29.910 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-01T04:13:29.966 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-11-01T04:13:29.993 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-11-01T04:13:30.024 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-11-01T04:13:30.051 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-google-auth. 2023-11-01T04:13:30.078 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-11-01T04:13:30.106 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-joblib. 2023-11-01T04:13:30.117 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-11-01T04:13:30.134 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-11-01T04:13:30.154 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package jq. 2023-11-01T04:13:30.169 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-11-01T04:13:30.173 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-11-01T04:13:30.212 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-11-01T04:13:30.335 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-sklearn. 2023-11-01T04:13:30.362 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-11-01T04:13:30.401 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-01T04:13:30.437 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-11-01T04:13:30.439 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package socat. 2023-11-01T04:13:30.455 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-11-01T04:13:30.463 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-11-01T04:13:30.503 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-01T04:13:30.535 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-11-01T04:13:30.562 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-11-01T04:13:30.601 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-11-01T04:13:30.646 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-11-01T04:13:30.913 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-sklearn. 2023-11-01T04:13:30.916 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-websocket. 2023-11-01T04:13:30.939 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-11-01T04:13:30.944 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-11-01T04:13:30.979 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-01T04:13:30.982 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-11-01T04:13:31.009 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package xmlstarlet. 2023-11-01T04:13:31.034 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-11-01T04:13:31.075 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-11-01T04:13:31.077 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-11-01T04:13:31.105 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:31.143 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:31.311 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-kubernetes. 2023-11-01T04:13:31.338 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-11-01T04:13:31.377 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-01T04:13:31.430 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-test. 2023-11-01T04:13:31.458 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:31.496 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-test (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:31.637 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-11-01T04:13:31.665 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:31.704 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:32.008 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-11-01T04:13:32.024 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:32.066 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:32.088 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-cachetools. 2023-11-01T04:13:32.116 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-11-01T04:13:32.154 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-11-01T04:13:32.385 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libonig5:amd64. 2023-11-01T04:13:32.413 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-11-01T04:13:32.432 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pyasn1. 2023-11-01T04:13:32.448 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-11-01T04:13:32.452 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-01T04:13:32.482 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-11-01T04:13:32.673 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cachetools. 2023-11-01T04:13:32.700 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-11-01T04:13:32.740 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-11-01T04:13:32.754 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libjq1:amd64. 2023-11-01T04:13:32.782 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-11-01T04:13:32.785 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-11-01T04:13:32.813 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-11-01T04:13:32.821 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-01T04:13:32.843 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-11-01T04:13:33.042 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pyasn1. 2023-11-01T04:13:33.048 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package jq. 2023-11-01T04:13:33.064 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-11-01T04:13:33.070 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-11-01T04:13:33.109 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-11-01T04:13:33.115 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-11-01T04:13:33.163 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-rsa. 2023-11-01T04:13:33.191 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-11-01T04:13:33.221 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-rsa (4.8-1) ... 2023-11-01T04:13:33.368 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package socat. 2023-11-01T04:13:33.384 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-11-01T04:13:33.426 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-11-01T04:13:33.462 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-11-01T04:13:33.489 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-11-01T04:13:33.529 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-11-01T04:13:33.550 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-google-auth. 2023-11-01T04:13:33.577 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-11-01T04:13:33.616 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-11-01T04:13:33.737 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package xmlstarlet. 2023-11-01T04:13:33.750 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-volume. 2023-11-01T04:13:33.764 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-11-01T04:13:33.778 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:33.804 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-11-01T04:13:33.816 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-volume (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:33.840 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rsa. 2023-11-01T04:13:33.867 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-11-01T04:13:33.906 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rsa (4.8-1) ... 2023-11-01T04:13:33.919 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-11-01T04:13:33.947 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-11-01T04:13:33.986 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-01T04:13:34.112 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libcephfs-dev. 2023-11-01T04:13:34.128 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:34.166 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-test. 2023-11-01T04:13:34.170 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libcephfs-dev (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:34.182 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:34.224 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-test (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:34.235 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-google-auth. 2023-11-01T04:13:34.262 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-11-01T04:13:34.301 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-11-01T04:13:34.382 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-websocket. 2023-11-01T04:13:34.410 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-11-01T04:13:34.449 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-11-01T04:13:34.490 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package nvme-cli. 2023-11-01T04:13:34.518 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-11-01T04:13:34.548 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-11-01T04:13:34.621 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-11-01T04:13:34.648 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-11-01T04:13:34.687 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-01T04:13:34.712 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-kubernetes. 2023-11-01T04:13:34.740 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-11-01T04:13:34.779 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-01T04:13:34.871 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-11-01T04:13:34.898 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-11-01T04:13:34.937 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-11-01T04:13:35.084 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-websocket. 2023-11-01T04:13:35.111 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-11-01T04:13:35.150 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-11-01T04:13:35.340 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-iniconfig. 2023-11-01T04:13:35.368 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-11-01T04:13:35.406 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-11-01T04:13:35.434 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-11-01T04:13:35.450 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:35.492 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:35.495 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-kubernetes. 2023-11-01T04:13:35.522 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-11-01T04:13:35.562 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-01T04:13:35.709 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pastescript. 2023-11-01T04:13:35.737 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-11-01T04:13:35.775 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-11-01T04:13:35.820 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libonig5:amd64. 2023-11-01T04:13:35.848 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-11-01T04:13:35.886 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-01T04:13:36.138 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pluggy. 2023-11-01T04:13:36.165 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-11-01T04:13:36.173 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libjq1:amd64. 2023-11-01T04:13:36.200 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-11-01T04:13:36.204 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-11-01T04:13:36.231 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-01T04:13:36.467 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-11-01T04:13:36.483 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:36.495 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-volume. 2023-11-01T04:13:36.515 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-psutil. 2023-11-01T04:13:36.522 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:36.525 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:36.542 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package jq. 2023-11-01T04:13:36.543 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-11-01T04:13:36.561 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-volume (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:36.569 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-11-01T04:13:36.582 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-11-01T04:13:36.609 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-11-01T04:13:36.824 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libcephfs-dev. 2023-11-01T04:13:36.836 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libonig5:amd64. 2023-11-01T04:13:36.839 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:36.863 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-11-01T04:13:36.870 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package socat. 2023-11-01T04:13:36.890 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libcephfs-dev (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:36.897 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-11-01T04:13:36.903 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-01T04:13:36.935 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-py. 2023-11-01T04:13:36.936 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-11-01T04:13:36.962 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-11-01T04:13:37.009 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-py (1.10.0-1) ... 2023-11-01T04:13:37.248 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libjq1:amd64. 2023-11-01T04:13:37.256 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package xmlstarlet. 2023-11-01T04:13:37.260 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package nvme-cli. 2023-11-01T04:13:37.275 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-11-01T04:13:37.284 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-11-01T04:13:37.287 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-11-01T04:13:37.314 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-01T04:13:37.322 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-11-01T04:13:37.326 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-11-01T04:13:37.330 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pygments. 2023-11-01T04:13:37.357 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-11-01T04:13:37.396 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-11-01T04:13:37.583 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package jq. 2023-11-01T04:13:37.599 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-11-01T04:13:37.641 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-11-01T04:13:37.673 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-11-01T04:13:37.676 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-test. 2023-11-01T04:13:37.692 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:37.700 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-11-01T04:13:37.735 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-test (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:37.739 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-11-01T04:13:37.869 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package socat. 2023-11-01T04:13:37.883 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pyinotify. 2023-11-01T04:13:37.896 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-11-01T04:13:37.912 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-11-01T04:13:37.927 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-11-01T04:13:37.949 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-11-01T04:13:38.176 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-iniconfig. 2023-11-01T04:13:38.204 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-11-01T04:13:38.230 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package xmlstarlet. 2023-11-01T04:13:38.242 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-11-01T04:13:38.244 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-toml. 2023-11-01T04:13:38.246 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-11-01T04:13:38.272 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-11-01T04:13:38.280 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-11-01T04:13:38.310 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-11-01T04:13:38.545 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pastescript. 2023-11-01T04:13:38.573 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-11-01T04:13:38.611 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-11-01T04:13:38.613 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pytest. 2023-11-01T04:13:38.626 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-test. 2023-11-01T04:13:38.641 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-11-01T04:13:38.642 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:38.679 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-11-01T04:13:38.684 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-test (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:38.957 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pluggy. 2023-11-01T04:13:38.985 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-11-01T04:13:39.023 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-11-01T04:13:39.041 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-simplejson. 2023-11-01T04:13:39.069 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-11-01T04:13:39.108 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-11-01T04:13:39.335 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-psutil. 2023-11-01T04:13:39.360 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-11-01T04:13:39.362 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-11-01T04:13:39.388 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-11-01T04:13:39.401 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-11-01T04:13:39.427 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-11-01T04:13:39.721 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-py. 2023-11-01T04:13:39.748 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-11-01T04:13:39.778 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-py (1.10.0-1) ... 2023-11-01T04:13:40.040 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pygments. 2023-11-01T04:13:40.068 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-11-01T04:13:40.098 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-11-01T04:13:40.156 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package radosgw. 2023-11-01T04:13:40.184 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../107-radosgw_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:40.222 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking radosgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:40.247 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-volume. 2023-11-01T04:13:40.275 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:40.314 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-volume (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:40.569 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyinotify. 2023-11-01T04:13:40.597 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-11-01T04:13:40.601 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libcephfs-dev. 2023-11-01T04:13:40.628 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:40.635 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-11-01T04:13:40.667 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libcephfs-dev (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:40.946 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-toml. 2023-11-01T04:13:40.974 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-11-01T04:13:41.013 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-11-01T04:13:41.013 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-volume. 2023-11-01T04:13:41.040 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6776-g420cd129-1jammy_all.deb ... 2023-11-01T04:13:41.046 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package nvme-cli. 2023-11-01T04:13:41.073 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-11-01T04:13:41.079 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-volume (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:41.084 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package rbd-fuse. 2023-11-01T04:13:41.112 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-11-01T04:13:41.112 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:41.208 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking rbd-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:41.315 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pytest. 2023-11-01T04:13:41.343 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-11-01T04:13:41.382 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-11-01T04:13:41.383 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libcephfs-dev. 2023-11-01T04:13:41.399 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:41.441 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libcephfs-dev (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:41.459 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-11-01T04:13:41.475 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-11-01T04:13:41.504 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-11-01T04:13:41.517 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-11-01T04:13:41.736 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-simplejson. 2023-11-01T04:13:41.753 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package nvme-cli. 2023-11-01T04:13:41.755 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-01T04:13:41.763 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-11-01T04:13:41.780 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-11-01T04:13:41.802 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-11-01T04:13:41.811 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-11-01T04:13:41.881 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-11-01T04:13:41.937 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-iniconfig. 2023-11-01T04:13:41.964 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-11-01T04:13:42.003 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-11-01T04:13:42.071 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-11-01T04:13:42.098 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-11-01T04:13:42.124 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-11-01T04:13:42.137 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-11-01T04:13:42.138 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-11-01T04:13:42.152 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-11-01T04:13:42.191 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-11-01T04:13:42.306 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pastescript. 2023-11-01T04:13:42.333 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-11-01T04:13:42.372 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-11-01T04:13:42.495 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-11-01T04:13:42.569 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-iniconfig. 2023-11-01T04:13:42.585 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-11-01T04:13:42.627 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-11-01T04:13:42.667 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pluggy. 2023-11-01T04:13:42.695 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-11-01T04:13:42.725 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-11-01T04:13:42.858 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package radosgw. 2023-11-01T04:13:42.886 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../107-radosgw_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:42.924 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking radosgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:42.930 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pastescript. 2023-11-01T04:13:42.957 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-11-01T04:13:42.985 INFO:teuthology.orchestra.run.smithi181.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-11-01T04:13:42.996 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-11-01T04:13:43.004 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-psutil. 2023-11-01T04:13:43.031 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-11-01T04:13:43.058 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:43.087 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-11-01T04:13:43.300 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pluggy. 2023-11-01T04:13:43.327 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-11-01T04:13:43.384 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-11-01T04:13:43.406 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-py. 2023-11-01T04:13:43.434 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-11-01T04:13:43.473 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-py (1.10.0-1) ... 2023-11-01T04:13:43.625 INFO:teuthology.orchestra.run.smithi181.stdout:Adding system user cephadm....done 2023-11-01T04:13:43.702 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-psutil. 2023-11-01T04:13:43.730 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-11-01T04:13:43.734 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pygments. 2023-11-01T04:13:43.747 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-01T04:13:43.750 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-11-01T04:13:43.768 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-11-01T04:13:43.784 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-11-01T04:13:43.927 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package rbd-fuse. 2023-11-01T04:13:43.955 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:43.993 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking rbd-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:44.040 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-11-01T04:13:44.113 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-py. 2023-11-01T04:13:44.140 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-11-01T04:13:44.179 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-py (1.10.0-1) ... 2023-11-01T04:13:44.222 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pyinotify. 2023-11-01T04:13:44.249 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-11-01T04:13:44.288 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-11-01T04:13:44.298 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-11-01T04:13:44.308 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-11-01T04:13:44.409 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-11-01T04:13:44.516 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pygments. 2023-11-01T04:13:44.543 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-11-01T04:13:44.548 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-01T04:13:44.582 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-11-01T04:13:44.590 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-toml. 2023-11-01T04:13:44.618 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-11-01T04:13:44.657 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-11-01T04:13:44.660 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-11-01T04:13:44.683 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-11-01T04:13:44.920 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-11-01T04:13:44.941 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-11-01T04:13:44.985 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pytest. 2023-11-01T04:13:45.012 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-11-01T04:13:45.028 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pyinotify. 2023-11-01T04:13:45.045 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-py (1.10.0-1) ... 2023-11-01T04:13:45.051 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-11-01T04:13:45.052 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-11-01T04:13:45.086 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-11-01T04:13:45.268 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-11-01T04:13:45.389 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-toml. 2023-11-01T04:13:45.397 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-11-01T04:13:45.405 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-simplejson. 2023-11-01T04:13:45.416 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-11-01T04:13:45.432 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-11-01T04:13:45.455 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-11-01T04:13:45.471 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-11-01T04:13:45.708 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pytest. 2023-11-01T04:13:45.724 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-11-01T04:13:45.739 INFO:teuthology.orchestra.run.smithi190.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-11-01T04:13:45.749 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-11-01T04:13:45.757 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-11-01T04:13:45.776 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-11-01T04:13:45.791 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-11-01T04:13:45.815 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-11-01T04:13:45.843 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:46.033 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-11-01T04:13:46.095 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-simplejson. 2023-11-01T04:13:46.123 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-11-01T04:13:46.161 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-11-01T04:13:46.343 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-11-01T04:13:46.395 INFO:teuthology.orchestra.run.smithi190.stdout:Adding system user cephadm....done 2023-11-01T04:13:46.430 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-11-01T04:13:46.446 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-11-01T04:13:46.488 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-11-01T04:13:46.511 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package radosgw. 2023-11-01T04:13:46.538 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../107-radosgw_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:46.557 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-01T04:13:46.577 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking radosgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:46.611 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-11-01T04:13:46.833 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-11-01T04:13:46.870 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-11-01T04:13:47.093 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-11-01T04:13:47.129 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-ceph-argparse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:47.167 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package radosgw. 2023-11-01T04:13:47.195 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../107-radosgw_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:47.219 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-11-01T04:13:47.234 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking radosgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:47.355 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package rbd-fuse. 2023-11-01T04:13:47.371 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:47.413 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking rbd-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:47.470 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-11-01T04:13:47.489 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-01T04:13:47.665 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-11-01T04:13:47.717 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-11-01T04:13:47.721 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-11-01T04:13:47.822 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-py (1.10.0-1) ... 2023-11-01T04:13:47.924 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-01T04:13:47.970 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package rbd-fuse. 2023-11-01T04:13:47.976 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-01T04:13:47.986 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6776-g420cd129-1jammy_amd64.deb ... 2023-11-01T04:13:48.020 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking rbd-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:48.050 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-11-01T04:13:48.132 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-11-01T04:13:48.136 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-11-01T04:13:48.298 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-11-01T04:13:48.424 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-11-01T04:13:48.427 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-01T04:13:48.517 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-11-01T04:13:48.549 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-11-01T04:13:48.552 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-11-01T04:13:48.650 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-11-01T04:13:48.768 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-11-01T04:13:48.784 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-11-01T04:13:48.829 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-11-01T04:13:48.886 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-11-01T04:13:49.061 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-11-01T04:13:49.105 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-11-01T04:13:49.241 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-11-01T04:13:49.250 INFO:teuthology.orchestra.run.smithi173.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-11-01T04:13:49.312 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-11-01T04:13:49.321 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:49.423 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-11-01T04:13:49.579 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-11-01T04:13:49.720 INFO:teuthology.orchestra.run.smithi153.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-11-01T04:13:49.794 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:49.805 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-ceph-argparse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:49.898 INFO:teuthology.orchestra.run.smithi173.stdout:Adding system user cephadm....done 2023-11-01T04:13:50.061 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-01T04:13:50.066 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-11-01T04:13:50.111 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-01T04:13:50.220 INFO:teuthology.orchestra.run.smithi153.stdout:Adding system user cephadm....done 2023-11-01T04:13:50.299 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-11-01T04:13:50.326 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-11-01T04:13:50.329 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-11-01T04:13:50.425 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-11-01T04:13:50.452 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-11-01T04:13:50.542 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-01T04:13:50.563 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-11-01T04:13:50.578 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-01T04:13:50.668 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-11-01T04:13:50.672 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-11-01T04:13:50.693 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-11-01T04:13:50.887 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-11-01T04:13:50.923 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-11-01T04:13:50.944 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-11-01T04:13:51.027 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-01T04:13:51.087 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-11-01T04:13:51.153 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-11-01T04:13:51.183 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-11-01T04:13:51.305 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-11-01T04:13:51.326 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-py (1.10.0-1) ... 2023-11-01T04:13:51.337 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-11-01T04:13:51.412 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-11-01T04:13:51.628 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-11-01T04:13:51.655 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-11-01T04:13:51.698 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-11-01T04:13:51.822 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-11-01T04:13:51.965 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-11-01T04:13:51.971 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-11-01T04:13:52.006 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-py (1.10.0-1) ... 2023-11-01T04:13:52.108 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-01T04:13:52.222 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-11-01T04:13:52.299 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-11-01T04:13:52.392 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-01T04:13:52.489 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-11-01T04:13:52.608 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-11-01T04:13:52.660 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-11-01T04:13:52.727 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-11-01T04:13:52.741 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-11-01T04:13:52.868 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-11-01T04:13:52.919 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-11-01T04:13:52.986 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-11-01T04:13:52.994 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-11-01T04:13:53.000 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-11-01T04:13:53.113 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:53.145 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-01T04:13:53.187 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-11-01T04:13:53.268 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-ceph-argparse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:53.414 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-idna (3.3-1) ... 2023-11-01T04:13:53.438 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-11-01T04:13:53.446 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-11-01T04:13:53.549 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-01T04:13:53.714 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-11-01T04:13:53.736 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-11-01T04:13:53.748 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-01T04:13:53.822 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-11-01T04:13:53.965 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-ceph-argparse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:53.988 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-01T04:13:54.048 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-11-01T04:13:54.097 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-11-01T04:13:54.257 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-11-01T04:13:54.359 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-11-01T04:13:54.416 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-01T04:13:54.464 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-01T04:13:54.590 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-11-01T04:13:54.601 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-11-01T04:13:54.701 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-01T04:13:54.727 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-11-01T04:13:54.858 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-11-01T04:13:54.862 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:13:55.013 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-toml (0.10.2-1) ... 2023-11-01T04:13:55.052 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-11-01T04:13:55.104 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-11-01T04:13:55.126 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-11-01T04:13:55.229 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-11-01T04:13:55.297 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-01T04:13:55.303 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-11-01T04:13:55.424 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-11-01T04:13:55.427 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-11-01T04:13:55.429 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:55.497 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-11-01T04:13:55.533 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-11-01T04:13:55.714 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-idna (3.3-1) ... 2023-11-01T04:13:55.748 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-11-01T04:13:55.817 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-11-01T04:13:56.006 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-01T04:13:56.033 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-11-01T04:13:56.054 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-11-01T04:13:56.077 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-11-01T04:13:56.305 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-11-01T04:13:56.328 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:13:56.456 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-11-01T04:13:56.471 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-11-01T04:13:56.582 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-01T04:13:56.622 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-11-01T04:13:56.676 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-11-01T04:13:56.875 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-11-01T04:13:56.927 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-11-01T04:13:56.948 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-11-01T04:13:57.087 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-11-01T04:13:57.145 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:13:57.208 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-11-01T04:13:57.213 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-11-01T04:13:57.254 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-toml (0.10.2-1) ... 2023-11-01T04:13:57.259 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-11-01T04:13:57.450 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-11-01T04:13:57.497 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-01T04:13:57.498 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-11-01T04:13:57.502 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-11-01T04:13:57.598 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-11-01T04:13:57.701 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-11-01T04:13:57.707 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-11-01T04:13:57.857 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-11-01T04:13:57.862 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-01T04:13:57.952 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-11-01T04:13:57.992 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-11-01T04:13:58.091 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-11-01T04:13:58.147 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-01T04:13:58.220 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-01T04:13:58.252 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-11-01T04:13:58.371 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-11-01T04:13:58.460 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-11-01T04:13:58.502 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:13:58.514 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-11-01T04:13:58.628 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-11-01T04:13:58.673 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-01T04:13:58.729 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-11-01T04:13:58.753 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-11-01T04:13:58.765 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-11-01T04:13:58.815 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-01T04:13:58.891 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:59.022 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-11-01T04:13:59.087 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-11-01T04:13:59.091 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-11-01T04:13:59.193 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-idna (3.3-1) ... 2023-11-01T04:13:59.338 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-11-01T04:13:59.376 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-11-01T04:13:59.431 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-11-01T04:13:59.447 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:13:59.502 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-01T04:13:59.636 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-11-01T04:13:59.740 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-idna (3.3-1) ... 2023-11-01T04:13:59.775 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-11-01T04:14:00.025 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-11-01T04:14:00.033 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-11-01T04:14:00.036 INFO:teuthology.orchestra.run.smithi181.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-11-01T04:14:00.280 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-01T04:14:00.301 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-11-01T04:14:00.422 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-11-01T04:14:00.577 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-01T04:14:00.633 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:14:00.703 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-11-01T04:14:00.767 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-toml (0.10.2-1) ... 2023-11-01T04:14:00.841 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-11-01T04:14:00.971 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-01T04:14:01.018 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-01T04:14:01.089 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-01T04:14:01.100 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-portend (3.0.0-1) ... 2023-11-01T04:14:01.128 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-11-01T04:14:01.197 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:14:01.254 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-11-01T04:14:01.331 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-toml (0.10.2-1) ... 2023-11-01T04:14:01.360 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:14:01.373 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-11-01T04:14:01.486 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-11-01T04:14:01.555 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-11-01T04:14:01.565 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-11-01T04:14:01.666 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-11-01T04:14:01.700 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-11-01T04:14:01.806 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-11-01T04:14:01.812 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-11-01T04:14:01.826 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-11-01T04:14:01.925 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-11-01T04:14:02.065 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:14:02.113 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-01T04:14:02.127 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-11-01T04:14:02.191 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-11-01T04:14:02.317 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-11-01T04:14:02.319 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-11-01T04:14:02.387 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-11-01T04:14:02.406 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-11-01T04:14:02.532 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-rsa (4.8-1) ... 2023-11-01T04:14:02.562 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-01T04:14:02.594 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-11-01T04:14:02.638 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:14:02.688 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-11-01T04:14:02.755 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-11-01T04:14:02.800 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-11-01T04:14:02.828 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-11-01T04:14:02.856 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-11-01T04:14:03.067 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-01T04:14:03.071 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-11-01T04:14:03.089 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-11-01T04:14:03.133 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-11-01T04:14:03.330 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-11-01T04:14:03.335 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-11-01T04:14:03.348 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-portend (3.0.0-1) ... 2023-11-01T04:14:03.378 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-11-01T04:14:03.581 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-11-01T04:14:03.583 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:14:03.645 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-11-01T04:14:03.709 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-11-01T04:14:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-11-01T04:14:03.849 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-01T04:14:03.904 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-11-01T04:14:03.975 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-11-01T04:14:04.033 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-11-01T04:14:04.035 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-11-01T04:14:04.164 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-11-01T04:14:04.251 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-01T04:14:04.336 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-01T04:14:04.369 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-01T04:14:04.381 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-01T04:14:04.440 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-11-01T04:14:04.507 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up librados2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:04.541 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-11-01T04:14:04.612 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-11-01T04:14:04.634 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up librgw2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:04.645 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-11-01T04:14:04.738 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rsa (4.8-1) ... 2023-11-01T04:14:04.768 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:04.800 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-11-01T04:14:04.894 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libcephfs2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:04.910 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-11-01T04:14:04.929 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-11-01T04:14:05.005 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-11-01T04:14:05.020 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libradosstriper1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:05.180 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-11-01T04:14:05.195 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-11-01T04:14:05.257 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-11-01T04:14:05.265 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-01T04:14:05.487 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-11-01T04:14:05.516 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-11-01T04:14:05.524 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up librbd1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:05.562 INFO:teuthology.orchestra.run.smithi173.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-11-01T04:14:05.626 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-mgr-modules-core (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:05.738 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-11-01T04:14:05.760 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:05.842 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-01T04:14:05.892 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-11-01T04:14:05.959 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-11-01T04:14:05.977 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-01T04:14:05.977 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-01T04:14:06.096 INFO:teuthology.orchestra.run.smithi153.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-11-01T04:14:06.177 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-11-01T04:14:06.333 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-11-01T04:14:06.369 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-11-01T04:14:06.433 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-11-01T04:14:06.503 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libcephfs-dev (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:06.512 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-01T04:14:06.612 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-portend (3.0.0-1) ... 2023-11-01T04:14:06.629 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-rados (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:06.638 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librados2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:06.755 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-01T04:14:06.764 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librgw2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:06.819 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-11-01T04:14:06.863 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:14:06.874 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:06.983 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libcephfs2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:06.989 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-11-01T04:14:07.078 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-portend (3.0.0-1) ... 2023-11-01T04:14:07.109 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libradosstriper1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:07.315 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-11-01T04:14:07.320 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-11-01T04:14:07.396 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-11-01T04:14:07.522 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-11-01T04:14:07.596 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librbd1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:07.624 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-01T04:14:07.714 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-modules-core (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:07.823 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-11-01T04:14:07.849 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:07.900 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-11-01T04:14:07.993 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-rbd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:08.026 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-rsa (4.8-1) ... 2023-11-01T04:14:08.059 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-01T04:14:08.060 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-01T04:14:08.115 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-11-01T04:14:08.119 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up rbd-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:08.245 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-rgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:08.294 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-11-01T04:14:08.383 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-11-01T04:14:08.396 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-cephfs (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:08.484 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rsa (4.8-1) ... 2023-11-01T04:14:08.547 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:08.558 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libcephfs-dev (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:08.562 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-01T04:14:08.684 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rados (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:08.760 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-11-01T04:14:08.810 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-01T04:14:08.813 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-11-01T04:14:09.037 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-11-01T04:14:09.107 INFO:teuthology.orchestra.run.smithi181.stdout:Adding group ceph....done 2023-11-01T04:14:09.198 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-11-01T04:14:09.304 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-11-01T04:14:09.491 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-11-01T04:14:09.500 INFO:teuthology.orchestra.run.smithi181.stdout:Adding system user ceph....done 2023-11-01T04:14:09.668 INFO:teuthology.orchestra.run.smithi181.stdout:Setting system user ceph properties....done 2023-11-01T04:14:09.680 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-11-01T04:14:09.684 INFO:teuthology.orchestra.run.smithi181.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-11-01T04:14:09.817 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-11-01T04:14:09.826 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-01T04:14:09.944 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up librados2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:09.973 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-11-01T04:14:10.056 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rbd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:10.061 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up librgw2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:10.150 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-11-01T04:14:10.187 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:10.191 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up rbd-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:10.300 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-11-01T04:14:10.313 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libcephfs2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:10.316 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:10.435 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librados2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:10.439 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libradosstriper1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:10.442 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cephfs (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:10.560 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:10.586 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librgw2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:10.642 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-11-01T04:14:10.673 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-test (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:10.712 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:10.774 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up radosgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:10.838 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libcephfs2 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:10.902 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up librbd1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:10.947 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libradosstriper1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:11.003 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr-modules-core (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:11.070 INFO:teuthology.orchestra.run.smithi190.stdout:Adding group ceph....done 2023-11-01T04:14:11.113 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:11.150 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-11-01T04:14:11.296 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-01T04:14:11.296 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-01T04:14:11.326 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-01T04:14:11.326 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-01T04:14:11.435 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librbd1 (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:11.471 INFO:teuthology.orchestra.run.smithi190.stdout:Adding system user ceph....done 2023-11-01T04:14:11.561 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-modules-core (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:11.640 INFO:teuthology.orchestra.run.smithi190.stdout:Setting system user ceph properties....done 2023-11-01T04:14:11.655 INFO:teuthology.orchestra.run.smithi190.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-11-01T04:14:11.687 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:11.792 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-11-01T04:14:11.826 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-base (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:11.889 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libcephfs-dev (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:11.901 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-01T04:14:11.901 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-11-01T04:14:12.065 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-rados (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:12.110 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-11-01T04:14:12.208 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-01T04:14:12.209 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-11-01T04:14:12.497 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libcephfs-dev (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:12.594 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-test (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:12.639 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rados (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:12.695 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-mds (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:12.695 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up radosgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:12.757 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-11-01T04:14:12.935 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-01T04:14:12.935 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-01T04:14:13.155 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-01T04:14:13.155 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-01T04:14:13.355 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-rbd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:13.463 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-mgr (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:13.481 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up rbd-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:13.607 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-rgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:13.664 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-base (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:13.687 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-01T04:14:13.687 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-01T04:14:13.733 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-cephfs (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:13.850 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:13.937 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rbd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:14.023 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-11-01T04:14:14.063 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up rbd-fuse (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:14.164 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-osd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:14.189 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:14.298 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cephfs (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:14.318 INFO:teuthology.orchestra.run.smithi173.stdout:Adding group ceph....done 2023-11-01T04:14:14.399 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-common (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:14.486 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-01T04:14:14.486 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-01T04:14:14.524 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mds (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:14.695 INFO:teuthology.orchestra.run.smithi173.stdout:Adding system user ceph....done 2023-11-01T04:14:14.799 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-01T04:14:14.800 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-01T04:14:14.889 INFO:teuthology.orchestra.run.smithi173.stdout:Setting system user ceph properties....done 2023-11-01T04:14:14.902 INFO:teuthology.orchestra.run.smithi173.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-11-01T04:14:14.918 INFO:teuthology.orchestra.run.smithi153.stdout:Adding group ceph....done 2023-11-01T04:14:15.016 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:15.039 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-11-01T04:14:15.144 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:15.228 INFO:teuthology.orchestra.run.smithi153.stdout:Adding system user ceph....done 2023-11-01T04:14:15.312 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-mon (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:15.342 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:15.387 INFO:teuthology.orchestra.run.smithi153.stdout:Setting system user ceph properties....done 2023-11-01T04:14:15.399 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-11-01T04:14:15.407 INFO:teuthology.orchestra.run.smithi153.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-11-01T04:14:15.540 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-01T04:14:15.541 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-01T04:14:15.544 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-11-01T04:14:15.561 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-01T04:14:15.562 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-01T04:14:15.919 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-test (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:15.947 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-11-01T04:14:16.028 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up radosgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:16.047 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-mgr-cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:16.052 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-osd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:16.223 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:16.354 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-01T04:14:16.355 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-01T04:14:16.462 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-mgr-dashboard (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:16.475 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-test (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:16.511 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-01T04:14:16.511 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-01T04:14:16.601 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up radosgw (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:16.658 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-volume (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:16.870 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:16.980 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-base (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:16.988 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:16.994 INFO:teuthology.orchestra.run.smithi181.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-11-01T04:14:17.122 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-01T04:14:17.122 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-11-01T04:14:17.138 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mon (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:17.336 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-11-01T04:14:17.353 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-01T04:14:17.353 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-01T04:14:17.687 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-base (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:17.848 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mds (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:17.882 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:18.018 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-11-01T04:14:18.033 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:18.090 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-01T04:14:18.090 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-01T04:14:18.159 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-dashboard (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:18.335 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-volume (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:18.408 INFO:teuthology.orchestra.run.smithi181.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-11-01T04:14:18.555 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mds (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:18.616 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:18.679 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-11-01T04:14:18.766 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-01T04:14:18.767 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-11-01T04:14:18.844 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-01T04:14:18.844 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-01T04:14:18.863 INFO:teuthology.orchestra.run.smithi181.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-01T04:14:18.866 DEBUG:teuthology.parallel:result is None 2023-11-01T04:14:19.290 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:19.376 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-osd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:19.487 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-01T04:14:19.488 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-11-01T04:14:19.674 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-01T04:14:19.675 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-01T04:14:19.763 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-11-01T04:14:20.041 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-osd (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:20.194 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:20.229 INFO:teuthology.orchestra.run.smithi190.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-01T04:14:20.232 DEBUG:teuthology.parallel:result is None 2023-11-01T04:14:20.304 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:20.352 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-01T04:14:20.353 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-11-01T04:14:20.454 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mon (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:20.666 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-01T04:14:20.666 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-01T04:14:20.918 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:21.044 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:21.164 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr-cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:21.195 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mon (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:21.307 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:21.409 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-01T04:14:21.409 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-11-01T04:14:21.433 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr-dashboard (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:21.584 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-volume (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:21.936 INFO:teuthology.orchestra.run.smithi173.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-11-01T04:14:21.979 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-cephadm (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:22.130 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:22.231 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-dashboard (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:22.357 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-volume (18.0.0-6776-g420cd129-1jammy) ... 2023-11-01T04:14:22.677 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-11-01T04:14:23.105 INFO:teuthology.orchestra.run.smithi173.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-11-01T04:14:23.560 INFO:teuthology.orchestra.run.smithi173.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-01T04:14:23.563 DEBUG:teuthology.parallel:result is None 2023-11-01T04:14:23.812 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-11-01T04:14:24.279 INFO:teuthology.orchestra.run.smithi153.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-11-01T04:14:24.282 DEBUG:teuthology.parallel:result is None 2023-11-01T04:14:24.282 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-01T04:14:24.430 DEBUG:teuthology.orchestra.run.smithi153:> dpkg-query -W -f '${Version}' ceph 2023-11-01T04:14:24.448 INFO:teuthology.orchestra.run.smithi153.stdout:18.0.0-6776-g420cd129-1jammy 2023-11-01T04:14:24.449 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6776-g420cd129-1jammy 2023-11-01T04:14:24.450 INFO:teuthology.task.install:The correct ceph version 18.0.0-6776-g420cd129-1jammy is installed. 2023-11-01T04:14:24.450 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-01T04:14:24.592 DEBUG:teuthology.orchestra.run.smithi173:> dpkg-query -W -f '${Version}' ceph 2023-11-01T04:14:24.612 INFO:teuthology.orchestra.run.smithi173.stdout:18.0.0-6776-g420cd129-1jammy 2023-11-01T04:14:24.612 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6776-g420cd129-1jammy 2023-11-01T04:14:24.613 INFO:teuthology.task.install:The correct ceph version 18.0.0-6776-g420cd129-1jammy is installed. 2023-11-01T04:14:24.613 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-01T04:14:24.754 DEBUG:teuthology.orchestra.run.smithi181:> dpkg-query -W -f '${Version}' ceph 2023-11-01T04:14:24.773 INFO:teuthology.orchestra.run.smithi181.stdout:18.0.0-6776-g420cd129-1jammy 2023-11-01T04:14:24.773 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6776-g420cd129-1jammy 2023-11-01T04:14:24.774 INFO:teuthology.task.install:The correct ceph version 18.0.0-6776-g420cd129-1jammy is installed. 2023-11-01T04:14:24.774 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=420cd129397301b1521ceb74e1bdbd9b855d2189 2023-11-01T04:14:24.924 DEBUG:teuthology.orchestra.run.smithi190:> dpkg-query -W -f '${Version}' ceph 2023-11-01T04:14:24.943 INFO:teuthology.orchestra.run.smithi190.stdout:18.0.0-6776-g420cd129-1jammy 2023-11-01T04:14:24.943 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6776-g420cd129-1jammy 2023-11-01T04:14:24.944 INFO:teuthology.task.install:The correct ceph version 18.0.0-6776-g420cd129-1jammy is installed. 2023-11-01T04:14:24.944 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-01T04:14:24.944 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:14:24.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-01T04:14:24.959 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:14:24.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-01T04:14:24.973 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:14:24.974 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-01T04:14:24.988 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:14:24.988 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-01T04:14:25.002 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-01T04:14:25.002 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:14:25.003 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2023-11-01T04:14:25.015 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-01T04:14:25.070 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:14:25.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2023-11-01T04:14:25.084 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-01T04:14:25.136 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:14:25.136 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2023-11-01T04:14:25.149 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-01T04:14:25.203 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:14:25.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2023-11-01T04:14:25.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-01T04:14:25.274 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-01T04:14:25.275 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:14:25.275 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-01T04:14:25.289 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-01T04:14:25.342 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:14:25.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-01T04:14:25.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-01T04:14:25.413 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:14:25.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-01T04:14:25.427 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-01T04:14:25.483 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:14:25.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-01T04:14:25.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-01T04:14:25.550 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-01T04:14:25.551 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:14:25.551 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2023-11-01T04:14:25.564 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-01T04:14:25.619 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:14:25.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2023-11-01T04:14:25.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-01T04:14:25.688 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:14:25.688 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2023-11-01T04:14:25.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-01T04:14:25.754 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:14:25.755 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2023-11-01T04:14:25.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-01T04:14:25.823 INFO:teuthology.run_tasks:Running task ceph... 2023-11-01T04:14:25.881 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-01T04:14:25.881 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /var/log/ceph 2023-11-01T04:14:25.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /var/log/ceph 2023-11-01T04:14:25.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /var/log/ceph 2023-11-01T04:14:25.889 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /var/log/ceph 2023-11-01T04:14:25.903 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-01T04:14:25.903 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-01T04:14:25.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-01T04:14:25.943 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-01T04:14:25.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-01T04:14:25.958 INFO:tasks.ceph:Creating extra log directories... 2023-11-01T04:14:25.959 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-01T04:14:25.992 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-01T04:14:26.003 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-01T04:14:26.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-01T04:14:26.031 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-01T04:14:26.031 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 1, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-01T04:14:26.031 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312', 'branch': 'DBG_maintain_prefix_itr_2', 'description': 'rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'email': None, 'first_in_suite': False, 'job_id': '7442312', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'DBG_maintain_prefix_itr_2'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'DBG_maintain_prefix_itr_2', 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189'}}, 'owner': 'scheduled_benhanokh@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 9009, 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'DBG_maintain_prefix_itr_2', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189', 'targets': {'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOAGLJhjDq8mvVkTG8Hw5tTgR0UZXjowuOsiYalT+QNbCXSyYSvs8LG7L/jZHCvB0sTKLuOqXRMsQl+LlBrIlOM=', 'smithi173.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKZi16MWKJndbPT3R1FdevTMb+dXFqXxHmJE/P6kQ+Gh9wecRp0CFuX/SllIWmHsP+6ZRYdXM51iHWKE79Ov4VI=', 'smithi181.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG6klwHOgaJO69D4LNOd3TcLmJkkJ3EdPRk+R27RAzf40b49HV2S9P8kZbO/WVyXSV3JNaS4kVTEr6Fr0WSh2BI=', 'smithi190.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCRm8aIzpGl2fqDE99OAe/4/JqhCP3sSy1lyMbrSiFzkg5qUKVPKtmAiues2krHh9LOorVPXIHYKqqHJZs9ZqpM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 1, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189', 'cluster': 'ceph'}}, {'thrashosds': {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-10-31_16:43:39', 'tube': 'smithi', 'user': 'benhanokh', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-11-01T04:14:26.032 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-01T04:14:26.044 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-01T04:14:26.055 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-01T04:14:26.061 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-01T04:14:26.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/run/ceph 2023-11-01T04:14:26.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/run/ceph 2023-11-01T04:14:26.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/run/ceph 2023-11-01T04:14:26.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/run/ceph 2023-11-01T04:14:26.122 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:14:26.122 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2023-11-01T04:14:26.151 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-01T04:14:26.152 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2023-11-01T04:14:26.199 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-01T04:14:26.200 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T04:14:26.200 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 889 Links: 1 2023-11-01T04:14:26.200 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T04:14:26.200 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2023-11-01 04:09:00.948449723 +0000 2023-11-01T04:14:26.200 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2023-11-01 04:09:00.604452792 +0000 2023-11-01T04:14:26.201 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2023-11-01 04:09:00.604452792 +0000 2023-11-01T04:14:26.201 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2023-11-01T04:14:26.201 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-01T04:14:26.258 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2023-11-01T04:14:26.258 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2023-11-01T04:14:26.259 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000360495 s, 1.4 MB/s 2023-11-01T04:14:26.260 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-01T04:14:26.309 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2023-11-01T04:14:26.355 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-01T04:14:26.356 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T04:14:26.356 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 904 Links: 1 2023-11-01T04:14:26.356 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T04:14:26.356 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2023-11-01 04:09:01.516444656 +0000 2023-11-01T04:14:26.356 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2023-11-01 04:09:01.180447654 +0000 2023-11-01T04:14:26.357 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2023-11-01 04:09:01.180447654 +0000 2023-11-01T04:14:26.357 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2023-11-01T04:14:26.357 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-01T04:14:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2023-11-01T04:14:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2023-11-01T04:14:26.410 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000442783 s, 1.2 MB/s 2023-11-01T04:14:26.411 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-01T04:14:26.461 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2023-11-01T04:14:26.507 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-01T04:14:26.508 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T04:14:26.508 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 921 Links: 1 2023-11-01T04:14:26.508 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T04:14:26.508 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2023-11-01 04:09:02.076439661 +0000 2023-11-01T04:14:26.508 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2023-11-01 04:09:01.736442693 +0000 2023-11-01T04:14:26.509 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2023-11-01 04:09:01.736442693 +0000 2023-11-01T04:14:26.509 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2023-11-01T04:14:26.509 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-01T04:14:26.561 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2023-11-01T04:14:26.562 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2023-11-01T04:14:26.562 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000326538 s, 1.6 MB/s 2023-11-01T04:14:26.563 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-01T04:14:26.612 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2023-11-01T04:14:26.659 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-01T04:14:26.659 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T04:14:26.660 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 935 Links: 1 2023-11-01T04:14:26.660 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T04:14:26.660 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2023-11-01 04:09:02.656434486 +0000 2023-11-01T04:14:26.660 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2023-11-01 04:09:02.312437555 +0000 2023-11-01T04:14:26.660 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2023-11-01 04:09:02.312437555 +0000 2023-11-01T04:14:26.661 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2023-11-01T04:14:26.661 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-01T04:14:26.714 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2023-11-01T04:14:26.714 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2023-11-01T04:14:26.715 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.00035189 s, 1.5 MB/s 2023-11-01T04:14:26.716 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-01T04:14:26.765 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2023-11-01T04:14:26.765 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:14:26.766 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2023-11-01T04:14:26.771 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-01T04:14:26.772 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2023-11-01T04:14:26.818 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-01T04:14:26.818 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T04:14:26.818 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 890 Links: 1 2023-11-01T04:14:26.818 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T04:14:26.819 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2023-11-01 04:09:00.532446576 +0000 2023-11-01T04:14:26.819 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2023-11-01 04:09:00.184449685 +0000 2023-11-01T04:14:26.819 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2023-11-01 04:09:00.184449685 +0000 2023-11-01T04:14:26.819 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: - 2023-11-01T04:14:26.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-01T04:14:26.872 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2023-11-01T04:14:26.872 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2023-11-01T04:14:26.872 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000442322 s, 1.2 MB/s 2023-11-01T04:14:26.873 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-01T04:14:26.923 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2023-11-01T04:14:26.969 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-01T04:14:26.969 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T04:14:26.970 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 909 Links: 1 2023-11-01T04:14:26.970 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T04:14:26.970 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2023-11-01 04:09:01.112441395 +0000 2023-11-01T04:14:26.970 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2023-11-01 04:09:00.784444326 +0000 2023-11-01T04:14:26.971 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2023-11-01 04:09:00.784444326 +0000 2023-11-01T04:14:26.971 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: - 2023-11-01T04:14:26.971 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-01T04:14:27.023 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2023-11-01T04:14:27.023 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2023-11-01T04:14:27.024 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000400276 s, 1.3 MB/s 2023-11-01T04:14:27.025 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-01T04:14:27.075 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2023-11-01T04:14:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-01T04:14:27.122 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T04:14:27.122 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 924 Links: 1 2023-11-01T04:14:27.122 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T04:14:27.123 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2023-11-01 04:09:01.720435964 +0000 2023-11-01T04:14:27.123 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2023-11-01 04:09:01.368439109 +0000 2023-11-01T04:14:27.123 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2023-11-01 04:09:01.368439109 +0000 2023-11-01T04:14:27.123 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: - 2023-11-01T04:14:27.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-01T04:14:27.177 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2023-11-01T04:14:27.177 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2023-11-01T04:14:27.177 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.00042411 s, 1.2 MB/s 2023-11-01T04:14:27.178 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-01T04:14:27.231 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2023-11-01T04:14:27.278 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-01T04:14:27.278 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T04:14:27.278 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 942 Links: 1 2023-11-01T04:14:27.279 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T04:14:27.279 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2023-11-01 04:09:02.332430499 +0000 2023-11-01T04:14:27.279 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2023-11-01 04:09:01.968433749 +0000 2023-11-01T04:14:27.279 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2023-11-01 04:09:01.968433749 +0000 2023-11-01T04:14:27.280 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: - 2023-11-01T04:14:27.280 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-01T04:14:27.332 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2023-11-01T04:14:27.332 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2023-11-01T04:14:27.332 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000339251 s, 1.5 MB/s 2023-11-01T04:14:27.333 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-01T04:14:27.383 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2023-11-01T04:14:27.383 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:14:27.383 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2023-11-01T04:14:27.390 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-01T04:14:27.391 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2023-11-01T04:14:27.436 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-01T04:14:27.437 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T04:14:27.437 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 891 Links: 1 2023-11-01T04:14:27.437 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T04:14:27.437 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2023-11-01 04:09:02.452955440 +0000 2023-11-01T04:14:27.438 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2023-11-01 04:09:02.076958798 +0000 2023-11-01T04:14:27.438 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2023-11-01 04:09:02.076958798 +0000 2023-11-01T04:14:27.438 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2023-11-01T04:14:27.438 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-01T04:14:27.490 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2023-11-01T04:14:27.491 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2023-11-01T04:14:27.491 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000354285 s, 1.4 MB/s 2023-11-01T04:14:27.492 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-01T04:14:27.541 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2023-11-01T04:14:27.587 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-01T04:14:27.588 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T04:14:27.588 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 902 Links: 1 2023-11-01T04:14:27.588 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T04:14:27.588 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2023-11-01 04:09:03.132949365 +0000 2023-11-01T04:14:27.589 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2023-11-01 04:09:02.700953224 +0000 2023-11-01T04:14:27.589 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2023-11-01 04:09:02.700953224 +0000 2023-11-01T04:14:27.589 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2023-11-01T04:14:27.589 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-01T04:14:27.642 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2023-11-01T04:14:27.642 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2023-11-01T04:14:27.642 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000423456 s, 1.2 MB/s 2023-11-01T04:14:27.643 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-01T04:14:27.693 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2023-11-01T04:14:27.739 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-01T04:14:27.740 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T04:14:27.740 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 927 Links: 1 2023-11-01T04:14:27.740 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T04:14:27.740 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2023-11-01 04:09:03.712944184 +0000 2023-11-01T04:14:27.740 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2023-11-01 04:09:03.352947400 +0000 2023-11-01T04:14:27.741 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2023-11-01 04:09:03.352947400 +0000 2023-11-01T04:14:27.741 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2023-11-01T04:14:27.741 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-01T04:14:27.793 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2023-11-01T04:14:27.793 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2023-11-01T04:14:27.794 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000315708 s, 1.6 MB/s 2023-11-01T04:14:27.795 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-01T04:14:27.845 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2023-11-01T04:14:27.896 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-01T04:14:27.896 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T04:14:27.896 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 936 Links: 1 2023-11-01T04:14:27.896 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T04:14:27.896 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2023-11-01 04:09:04.260939290 +0000 2023-11-01T04:14:27.897 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2023-11-01 04:09:03.944942113 +0000 2023-11-01T04:14:27.897 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2023-11-01 04:09:03.944942113 +0000 2023-11-01T04:14:27.897 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2023-11-01T04:14:27.897 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-01T04:14:27.950 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2023-11-01T04:14:27.950 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2023-11-01T04:14:27.951 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000387523 s, 1.3 MB/s 2023-11-01T04:14:27.952 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-01T04:14:28.001 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2023-11-01T04:14:28.001 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:14:28.001 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2023-11-01T04:14:28.008 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-01T04:14:28.008 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2023-11-01T04:14:28.055 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-01T04:14:28.055 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T04:14:28.056 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 884 Links: 1 2023-11-01T04:14:28.056 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T04:14:28.056 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2023-11-01 04:09:02.603183492 +0000 2023-11-01T04:14:28.056 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2023-11-01 04:09:02.599183528 +0000 2023-11-01T04:14:28.057 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2023-11-01 04:09:02.599183528 +0000 2023-11-01T04:14:28.057 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2023-11-01T04:14:28.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-01T04:14:28.109 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2023-11-01T04:14:28.109 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2023-11-01T04:14:28.110 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000390001 s, 1.3 MB/s 2023-11-01T04:14:28.111 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-01T04:14:28.160 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2023-11-01T04:14:28.207 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-01T04:14:28.207 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T04:14:28.207 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 898 Links: 1 2023-11-01T04:14:28.207 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T04:14:28.208 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2023-11-01 04:09:03.511175382 +0000 2023-11-01T04:14:28.208 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2023-11-01 04:09:03.187178276 +0000 2023-11-01T04:14:28.208 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2023-11-01 04:09:03.187178276 +0000 2023-11-01T04:14:28.208 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2023-11-01T04:14:28.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-01T04:14:28.261 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2023-11-01T04:14:28.261 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2023-11-01T04:14:28.261 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000421047 s, 1.2 MB/s 2023-11-01T04:14:28.262 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-01T04:14:28.312 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2023-11-01T04:14:28.359 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-01T04:14:28.359 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T04:14:28.359 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 917 Links: 1 2023-11-01T04:14:28.360 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T04:14:28.360 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2023-11-01 04:09:03.747173273 +0000 2023-11-01T04:14:28.360 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2023-11-01 04:09:03.743173310 +0000 2023-11-01T04:14:28.360 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2023-11-01 04:09:03.743173310 +0000 2023-11-01T04:14:28.360 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2023-11-01T04:14:28.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-01T04:14:28.413 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2023-11-01T04:14:28.413 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2023-11-01T04:14:28.413 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.00041763 s, 1.2 MB/s 2023-11-01T04:14:28.415 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-01T04:14:28.464 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2023-11-01T04:14:28.511 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-01T04:14:28.511 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T04:14:28.511 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 940 Links: 1 2023-11-01T04:14:28.512 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T04:14:28.512 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2023-11-01 04:09:04.679164950 +0000 2023-11-01T04:14:28.512 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2023-11-01 04:09:04.331168058 +0000 2023-11-01T04:14:28.512 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2023-11-01 04:09:04.331168058 +0000 2023-11-01T04:14:28.512 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2023-11-01T04:14:28.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-01T04:14:28.565 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2023-11-01T04:14:28.565 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2023-11-01T04:14:28.566 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000388384 s, 1.3 MB/s 2023-11-01T04:14:28.567 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-01T04:14:28.616 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2023-11-01T04:14:28.616 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2023-11-01T04:14:28.616 INFO:tasks.ceph:Generating config... 2023-11-01T04:14:28.618 INFO:tasks.ceph:[osd] debug monc = 20 2023-11-01T04:14:28.618 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2023-11-01T04:14:28.618 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2023-11-01T04:14:28.619 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-01T04:14:28.619 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-01T04:14:28.619 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-11-01T04:14:28.620 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-11-01T04:14:28.620 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-11-01T04:14:28.620 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-01T04:14:28.620 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-01T04:14:28.621 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-11-01T04:14:28.621 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-01T04:14:28.621 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-01T04:14:28.622 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-01T04:14:28.622 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-11-01T04:14:28.622 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2023-11-01T04:14:28.622 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-11-01T04:14:28.623 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-11-01T04:14:28.623 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-01T04:14:28.623 INFO:tasks.ceph:[osd] osd max backfills = 3 2023-11-01T04:14:28.624 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-11-01T04:14:28.624 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-11-01T04:14:28.624 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-11-01T04:14:28.624 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-01T04:14:28.625 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-11-01T04:14:28.625 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-11-01T04:14:28.625 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2023-11-01T04:14:28.626 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2023-11-01T04:14:28.626 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-11-01T04:14:28.626 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-11-01T04:14:28.626 INFO:tasks.ceph:[global] osd debug inject dispatch delay duration = 0.1 2023-11-01T04:14:28.627 INFO:tasks.ceph:[global] osd debug inject dispatch delay probability = 0.1 2023-11-01T04:14:28.627 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2023-11-01T04:14:28.627 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-01T04:14:28.628 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-01T04:14:28.628 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-01T04:14:28.628 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-01T04:14:28.628 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-01T04:14:28.629 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-11-01T04:14:28.629 INFO:tasks.ceph:Setting up mon.a... 2023-11-01T04:14:28.629 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-01T04:14:28.663 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/ceph.keyring 2023-11-01T04:14:28.683 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-01T04:14:28.760 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-01T04:14:28.814 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.153'), ('mon.b', '172.21.15.173'), ('mon.c', '172.21.15.181')] 2023-11-01T04:14:28.814 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.153,172.21.15.173,172.21.15.181', 'mon election default strategy': 1, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_object_clean_region_max_num_intervals': 1000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2023-11-01T04:14:28.815 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:14:28.815 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-01T04:14:28.859 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.153 --add b 172.21.15.173 --add c 172.21.15.181 --print /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:14:28.943 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:14:28.944 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: generated fsid 0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d 2023-11-01T04:14:28.944 INFO:teuthology.orchestra.run.smithi153.stdout:setting min_mon_release = pacific 2023-11-01T04:14:28.944 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 0 2023-11-01T04:14:28.944 INFO:teuthology.orchestra.run.smithi153.stdout:fsid 0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d 2023-11-01T04:14:28.945 INFO:teuthology.orchestra.run.smithi153.stdout:last_changed 2023-11-01T04:14:28.945761+0000 2023-11-01T04:14:28.945 INFO:teuthology.orchestra.run.smithi153.stdout:created 2023-11-01T04:14:28.945761+0000 2023-11-01T04:14:28.945 INFO:teuthology.orchestra.run.smithi153.stdout:min_mon_release 16 (pacific) 2023-11-01T04:14:28.945 INFO:teuthology.orchestra.run.smithi153.stdout:election_strategy: 1 2023-11-01T04:14:28.945 INFO:teuthology.orchestra.run.smithi153.stdout:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2023-11-01T04:14:28.946 INFO:teuthology.orchestra.run.smithi153.stdout:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2023-11-01T04:14:28.946 INFO:teuthology.orchestra.run.smithi153.stdout:2: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.c 2023-11-01T04:14:28.946 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-11-01T04:14:28.947 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-01T04:14:28.995 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d... 2023-11-01T04:14:28.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-01T04:14:29.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-01T04:14:29.043 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-01T04:14:29.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-01T04:14:29.069 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2023-11-01T04:14:29.069 INFO:teuthology.orchestra.run.smithi153.stdout: chdir = "" 2023-11-01T04:14:29.069 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-01T04:14:29.070 INFO:teuthology.orchestra.run.smithi153.stdout: auth supported = cephx 2023-11-01T04:14:29.070 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.070 INFO:teuthology.orchestra.run.smithi153.stdout: filestore xattr use omap = true 2023-11-01T04:14:29.070 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.071 INFO:teuthology.orchestra.run.smithi153.stdout: mon clock drift allowed = 1.000 2023-11-01T04:14:29.071 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.071 INFO:teuthology.orchestra.run.smithi153.stdout: osd crush chooseleaf type = 0 2023-11-01T04:14:29.071 INFO:teuthology.orchestra.run.smithi153.stdout: auth debug = true 2023-11-01T04:14:29.071 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.072 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on old message = true 2023-11-01T04:14:29.072 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on bug = true 2023-11-01T04:14:29.072 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.072 INFO:teuthology.orchestra.run.smithi153.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-01T04:14:29.073 INFO:teuthology.orchestra.run.smithi153.stdout: mon pg warn max object skew = 0 2023-11-01T04:14:29.073 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.073 INFO:teuthology.orchestra.run.smithi153.stdout: # disable pg_autoscaler by default for new pools 2023-11-01T04:14:29.073 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-01T04:14:29.073 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.074 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default size = 2 2023-11-01T04:14:29.074 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.074 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow primary affinity = true 2023-11-01T04:14:29.074 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow pg remap = true 2023-11-01T04:14:29.074 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on legacy crush tunables = false 2023-11-01T04:14:29.075 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on crush straw calc version zero = false 2023-11-01T04:14:29.075 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on no sortbitwise = false 2023-11-01T04:14:29.075 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on osd down out interval zero = false 2023-11-01T04:14:29.075 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on too few osds = false 2023-11-01T04:14:29.076 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-01T04:14:29.076 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-01T04:14:29.076 INFO:teuthology.orchestra.run.smithi153.stdout: mon_allow_pool_size_one = true 2023-11-01T04:14:29.076 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.076 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-01T04:14:29.077 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.077 INFO:teuthology.orchestra.run.smithi153.stdout: osd default data pool replay window = 5 2023-11-01T04:14:29.077 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.077 INFO:teuthology.orchestra.run.smithi153.stdout: mon allow pool delete = true 2023-11-01T04:14:29.078 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.078 INFO:teuthology.orchestra.run.smithi153.stdout: mon cluster log file level = debug 2023-11-01T04:14:29.078 INFO:teuthology.orchestra.run.smithi153.stdout: debug asserts on shutdown = true 2023-11-01T04:14:29.078 INFO:teuthology.orchestra.run.smithi153.stdout: mon health detail to clog = false 2023-11-01T04:14:29.078 INFO:teuthology.orchestra.run.smithi153.stdout: mon host = "172.21.15.153,172.21.15.173,172.21.15.181" 2023-11-01T04:14:29.079 INFO:teuthology.orchestra.run.smithi153.stdout: mon election default strategy = 1 2023-11-01T04:14:29.079 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug inject dispatch delay duration = 0.1 2023-11-01T04:14:29.079 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug inject dispatch delay probability = 0.1 2023-11-01T04:14:29.079 INFO:teuthology.orchestra.run.smithi153.stdout: osd_object_clean_region_max_num_intervals = 1000 2023-11-01T04:14:29.080 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = 0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d 2023-11-01T04:14:29.080 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.080 INFO:teuthology.orchestra.run.smithi153.stdout:[osd] 2023-11-01T04:14:29.080 INFO:teuthology.orchestra.run.smithi153.stdout: osd journal size = 100 2023-11-01T04:14:29.080 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.081 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub load threshold = 5.0 2023-11-01T04:14:29.081 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub max interval = 120 2023-11-01T04:14:29.081 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock profile = high_recovery_ops 2023-11-01T04:14:29.081 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.082 INFO:teuthology.orchestra.run.smithi153.stdout: osd recover clone overlap = true 2023-11-01T04:14:29.082 INFO:teuthology.orchestra.run.smithi153.stdout: osd recovery max chunk = 1048576 2023-11-01T04:14:29.082 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.082 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug shutdown = true 2023-11-01T04:14:29.083 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug op order = true 2023-11-01T04:14:29.083 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify stray on activate = true 2023-11-01T04:14:29.083 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug trim objects = true 2023-11-01T04:14:29.083 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.083 INFO:teuthology.orchestra.run.smithi153.stdout: osd open classes on start = true 2023-11-01T04:14:29.084 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug pg log writeout = true 2023-11-01T04:14:29.084 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.084 INFO:teuthology.orchestra.run.smithi153.stdout: osd deep scrub update digest min age = 30 2023-11-01T04:14:29.084 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.085 INFO:teuthology.orchestra.run.smithi153.stdout: osd map max advance = 10 2023-11-01T04:14:29.085 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.085 INFO:teuthology.orchestra.run.smithi153.stdout: journal zero on create = true 2023-11-01T04:14:29.085 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.085 INFO:teuthology.orchestra.run.smithi153.stdout: filestore ondisk finisher threads = 3 2023-11-01T04:14:29.086 INFO:teuthology.orchestra.run.smithi153.stdout: filestore apply finisher threads = 3 2023-11-01T04:14:29.086 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.086 INFO:teuthology.orchestra.run.smithi153.stdout: bdev debug aio = true 2023-11-01T04:14:29.086 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug misdirected ops = true 2023-11-01T04:14:29.087 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 20 2023-11-01T04:14:29.087 INFO:teuthology.orchestra.run.smithi153.stdout: bluefs allocator = hybrid 2023-11-01T04:14:29.087 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore allocator = hybrid 2023-11-01T04:14:29.087 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore block size = 96636764160 2023-11-01T04:14:29.087 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore fsck on mount = True 2023-11-01T04:14:29.088 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore zero block detection = True 2023-11-01T04:14:29.088 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluefs = 20 2023-11-01T04:14:29.088 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluestore = 20 2023-11-01T04:14:29.088 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2023-11-01T04:14:29.088 INFO:teuthology.orchestra.run.smithi153.stdout: debug osd = 20 2023-11-01T04:14:29.089 INFO:teuthology.orchestra.run.smithi153.stdout: debug rocksdb = 10 2023-11-01T04:14:29.089 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd backfillfull_ratio = 0.85 2023-11-01T04:14:29.089 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd full ratio = 0.9 2023-11-01T04:14:29.089 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd nearfull ratio = 0.8 2023-11-01T04:14:29.089 INFO:teuthology.orchestra.run.smithi153.stdout: osd blocked scrub grace period = 3600 2023-11-01T04:14:29.090 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug reject backfill probability = 0.3 2023-11-01T04:14:29.090 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify cached snaps = True 2023-11-01T04:14:29.090 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify missing on start = True 2023-11-01T04:14:29.090 INFO:teuthology.orchestra.run.smithi153.stdout: osd failsafe full ratio = 0.95 2023-11-01T04:14:29.091 INFO:teuthology.orchestra.run.smithi153.stdout: osd max backfills = 3 2023-11-01T04:14:29.091 INFO:teuthology.orchestra.run.smithi153.stdout: osd max markdown count = 1000 2023-11-01T04:14:29.091 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock override recovery settings = True 2023-11-01T04:14:29.091 INFO:teuthology.orchestra.run.smithi153.stdout: osd objectstore = bluestore 2023-11-01T04:14:29.091 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue = debug_random 2023-11-01T04:14:29.092 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue cut off = debug_random 2023-11-01T04:14:29.092 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub min interval = 60 2023-11-01T04:14:29.092 INFO:teuthology.orchestra.run.smithi153.stdout: osd shutdown pgref assert = True 2023-11-01T04:14:29.092 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.092 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr] 2023-11-01T04:14:29.093 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2023-11-01T04:14:29.093 INFO:teuthology.orchestra.run.smithi153.stdout: debug mgr = 20 2023-11-01T04:14:29.093 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2023-11-01T04:14:29.093 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2023-11-01T04:14:29.093 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2023-11-01T04:14:29.094 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2023-11-01T04:14:29.094 INFO:teuthology.orchestra.run.smithi153.stdout: mgr/telemetry/nag = false 2023-11-01T04:14:29.094 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.094 INFO:teuthology.orchestra.run.smithi153.stdout:[mon] 2023-11-01T04:14:29.094 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2023-11-01T04:14:29.095 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2023-11-01T04:14:29.095 INFO:teuthology.orchestra.run.smithi153.stdout: debug paxos = 20 2023-11-01T04:14:29.095 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2023-11-01T04:14:29.095 INFO:teuthology.orchestra.run.smithi153.stdout: mon data avail warn = 5 2023-11-01T04:14:29.096 INFO:teuthology.orchestra.run.smithi153.stdout: mon mgr mkfs grace = 240 2023-11-01T04:14:29.096 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2023-11-01T04:14:29.096 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd reporter subtree level = osd 2023-11-01T04:14:29.096 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd prime pg temp = true 2023-11-01T04:14:29.096 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2023-11-01T04:14:29.097 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.097 INFO:teuthology.orchestra.run.smithi153.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-01T04:14:29.097 INFO:teuthology.orchestra.run.smithi153.stdout: auth mon ticket ttl = 660 # 11m 2023-11-01T04:14:29.097 INFO:teuthology.orchestra.run.smithi153.stdout: auth service ticket ttl = 240 # 4m 2023-11-01T04:14:29.097 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.098 INFO:teuthology.orchestra.run.smithi153.stdout: # don't complain about insecure global_id in the test suite 2023-11-01T04:14:29.098 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-01T04:14:29.098 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-01T04:14:29.098 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.099 INFO:teuthology.orchestra.run.smithi153.stdout: # 1m isn't quite enough 2023-11-01T04:14:29.099 INFO:teuthology.orchestra.run.smithi153.stdout: mon_down_mkfs_grace = 2m 2023-11-01T04:14:29.099 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.099 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_filestore_osds = false 2023-11-01T04:14:29.100 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub interval = 300 2023-11-01T04:14:29.100 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:14:29.100 INFO:teuthology.orchestra.run.smithi153.stdout:[client] 2023-11-01T04:14:29.100 INFO:teuthology.orchestra.run.smithi153.stdout: rgw cache enabled = true 2023-11-01T04:14:29.100 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable ops log = true 2023-11-01T04:14:29.101 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable usage log = true 2023-11-01T04:14:29.101 INFO:teuthology.orchestra.run.smithi153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-01T04:14:29.101 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-01T04:14:29.101 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.a] 2023-11-01T04:14:29.101 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.b] 2023-11-01T04:14:29.102 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.c] 2023-11-01T04:14:29.104 INFO:teuthology.orchestra.run.smithi190.stdout:[global] 2023-11-01T04:14:29.104 INFO:teuthology.orchestra.run.smithi190.stdout: chdir = "" 2023-11-01T04:14:29.105 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-01T04:14:29.105 INFO:teuthology.orchestra.run.smithi190.stdout: auth supported = cephx 2023-11-01T04:14:29.105 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.105 INFO:teuthology.orchestra.run.smithi190.stdout: filestore xattr use omap = true 2023-11-01T04:14:29.105 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.106 INFO:teuthology.orchestra.run.smithi190.stdout: mon clock drift allowed = 1.000 2023-11-01T04:14:29.106 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.106 INFO:teuthology.orchestra.run.smithi190.stdout: osd crush chooseleaf type = 0 2023-11-01T04:14:29.106 INFO:teuthology.orchestra.run.smithi190.stdout: auth debug = true 2023-11-01T04:14:29.107 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.107 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on old message = true 2023-11-01T04:14:29.107 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on bug = true 2023-11-01T04:14:29.107 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.107 INFO:teuthology.orchestra.run.smithi190.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-01T04:14:29.107 INFO:teuthology.orchestra.run.smithi190.stdout: mon pg warn max object skew = 0 2023-11-01T04:14:29.108 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.108 INFO:teuthology.orchestra.run.smithi190.stdout: # disable pg_autoscaler by default for new pools 2023-11-01T04:14:29.108 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-01T04:14:29.108 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.108 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default size = 2 2023-11-01T04:14:29.109 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.109 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow primary affinity = true 2023-11-01T04:14:29.109 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow pg remap = true 2023-11-01T04:14:29.109 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on legacy crush tunables = false 2023-11-01T04:14:29.109 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on crush straw calc version zero = false 2023-11-01T04:14:29.110 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on no sortbitwise = false 2023-11-01T04:14:29.110 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on osd down out interval zero = false 2023-11-01T04:14:29.110 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on too few osds = false 2023-11-01T04:14:29.110 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-01T04:14:29.110 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-01T04:14:29.111 INFO:teuthology.orchestra.run.smithi190.stdout: mon_allow_pool_size_one = true 2023-11-01T04:14:29.111 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.111 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-01T04:14:29.111 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.111 INFO:teuthology.orchestra.run.smithi190.stdout: osd default data pool replay window = 5 2023-11-01T04:14:29.112 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.112 INFO:teuthology.orchestra.run.smithi190.stdout: mon allow pool delete = true 2023-11-01T04:14:29.112 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.112 INFO:teuthology.orchestra.run.smithi190.stdout: mon cluster log file level = debug 2023-11-01T04:14:29.112 INFO:teuthology.orchestra.run.smithi190.stdout: debug asserts on shutdown = true 2023-11-01T04:14:29.113 INFO:teuthology.orchestra.run.smithi190.stdout: mon health detail to clog = false 2023-11-01T04:14:29.113 INFO:teuthology.orchestra.run.smithi190.stdout: mon host = "172.21.15.153,172.21.15.173,172.21.15.181" 2023-11-01T04:14:29.113 INFO:teuthology.orchestra.run.smithi190.stdout: mon election default strategy = 1 2023-11-01T04:14:29.113 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug inject dispatch delay duration = 0.1 2023-11-01T04:14:29.113 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug inject dispatch delay probability = 0.1 2023-11-01T04:14:29.114 INFO:teuthology.orchestra.run.smithi190.stdout: osd_object_clean_region_max_num_intervals = 1000 2023-11-01T04:14:29.114 INFO:teuthology.orchestra.run.smithi190.stdout: fsid = 0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d 2023-11-01T04:14:29.114 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.114 INFO:teuthology.orchestra.run.smithi190.stdout:[osd] 2023-11-01T04:14:29.114 INFO:teuthology.orchestra.run.smithi190.stdout: osd journal size = 100 2023-11-01T04:14:29.115 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.115 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub load threshold = 5.0 2023-11-01T04:14:29.115 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub max interval = 120 2023-11-01T04:14:29.115 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock profile = high_recovery_ops 2023-11-01T04:14:29.115 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.116 INFO:teuthology.orchestra.run.smithi190.stdout: osd recover clone overlap = true 2023-11-01T04:14:29.116 INFO:teuthology.orchestra.run.smithi190.stdout: osd recovery max chunk = 1048576 2023-11-01T04:14:29.116 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.116 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug shutdown = true 2023-11-01T04:14:29.116 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug op order = true 2023-11-01T04:14:29.116 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify stray on activate = true 2023-11-01T04:14:29.117 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug trim objects = true 2023-11-01T04:14:29.117 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.117 INFO:teuthology.orchestra.run.smithi190.stdout: osd open classes on start = true 2023-11-01T04:14:29.117 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug pg log writeout = true 2023-11-01T04:14:29.117 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.118 INFO:teuthology.orchestra.run.smithi190.stdout: osd deep scrub update digest min age = 30 2023-11-01T04:14:29.118 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.118 INFO:teuthology.orchestra.run.smithi190.stdout: osd map max advance = 10 2023-11-01T04:14:29.118 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.118 INFO:teuthology.orchestra.run.smithi190.stdout: journal zero on create = true 2023-11-01T04:14:29.119 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.119 INFO:teuthology.orchestra.run.smithi190.stdout: filestore ondisk finisher threads = 3 2023-11-01T04:14:29.119 INFO:teuthology.orchestra.run.smithi190.stdout: filestore apply finisher threads = 3 2023-11-01T04:14:29.119 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.119 INFO:teuthology.orchestra.run.smithi190.stdout: bdev debug aio = true 2023-11-01T04:14:29.120 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug misdirected ops = true 2023-11-01T04:14:29.120 INFO:teuthology.orchestra.run.smithi190.stdout: debug monc = 20 2023-11-01T04:14:29.120 INFO:teuthology.orchestra.run.smithi190.stdout: bluefs allocator = hybrid 2023-11-01T04:14:29.120 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore allocator = hybrid 2023-11-01T04:14:29.120 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore block size = 96636764160 2023-11-01T04:14:29.121 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore fsck on mount = True 2023-11-01T04:14:29.121 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore zero block detection = True 2023-11-01T04:14:29.121 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluefs = 20 2023-11-01T04:14:29.121 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluestore = 20 2023-11-01T04:14:29.121 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2023-11-01T04:14:29.122 INFO:teuthology.orchestra.run.smithi190.stdout: debug osd = 20 2023-11-01T04:14:29.122 INFO:teuthology.orchestra.run.smithi190.stdout: debug rocksdb = 10 2023-11-01T04:14:29.122 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd backfillfull_ratio = 0.85 2023-11-01T04:14:29.122 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd full ratio = 0.9 2023-11-01T04:14:29.122 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd nearfull ratio = 0.8 2023-11-01T04:14:29.123 INFO:teuthology.orchestra.run.smithi190.stdout: osd blocked scrub grace period = 3600 2023-11-01T04:14:29.123 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug reject backfill probability = 0.3 2023-11-01T04:14:29.123 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify cached snaps = True 2023-11-01T04:14:29.123 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify missing on start = True 2023-11-01T04:14:29.123 INFO:teuthology.orchestra.run.smithi190.stdout: osd failsafe full ratio = 0.95 2023-11-01T04:14:29.123 INFO:teuthology.orchestra.run.smithi190.stdout: osd max backfills = 3 2023-11-01T04:14:29.124 INFO:teuthology.orchestra.run.smithi190.stdout: osd max markdown count = 1000 2023-11-01T04:14:29.124 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock override recovery settings = True 2023-11-01T04:14:29.124 INFO:teuthology.orchestra.run.smithi190.stdout: osd objectstore = bluestore 2023-11-01T04:14:29.124 INFO:teuthology.orchestra.run.smithi190.stdout: osd op queue = debug_random 2023-11-01T04:14:29.124 INFO:teuthology.orchestra.run.smithi190.stdout: osd op queue cut off = debug_random 2023-11-01T04:14:29.125 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub min interval = 60 2023-11-01T04:14:29.125 INFO:teuthology.orchestra.run.smithi190.stdout: osd shutdown pgref assert = True 2023-11-01T04:14:29.125 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.125 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr] 2023-11-01T04:14:29.125 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2023-11-01T04:14:29.126 INFO:teuthology.orchestra.run.smithi190.stdout: debug mgr = 20 2023-11-01T04:14:29.126 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2023-11-01T04:14:29.126 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2023-11-01T04:14:29.126 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2023-11-01T04:14:29.126 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2023-11-01T04:14:29.127 INFO:teuthology.orchestra.run.smithi190.stdout: mgr/telemetry/nag = false 2023-11-01T04:14:29.127 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.127 INFO:teuthology.orchestra.run.smithi190.stdout:[mon] 2023-11-01T04:14:29.127 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2023-11-01T04:14:29.127 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2023-11-01T04:14:29.128 INFO:teuthology.orchestra.run.smithi190.stdout: debug paxos = 20 2023-11-01T04:14:29.128 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2023-11-01T04:14:29.128 INFO:teuthology.orchestra.run.smithi190.stdout: mon data avail warn = 5 2023-11-01T04:14:29.128 INFO:teuthology.orchestra.run.smithi190.stdout: mon mgr mkfs grace = 240 2023-11-01T04:14:29.128 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2023-11-01T04:14:29.129 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd reporter subtree level = osd 2023-11-01T04:14:29.129 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd prime pg temp = true 2023-11-01T04:14:29.129 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2023-11-01T04:14:29.129 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.129 INFO:teuthology.orchestra.run.smithi190.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-01T04:14:29.129 INFO:teuthology.orchestra.run.smithi190.stdout: auth mon ticket ttl = 660 # 11m 2023-11-01T04:14:29.130 INFO:teuthology.orchestra.run.smithi190.stdout: auth service ticket ttl = 240 # 4m 2023-11-01T04:14:29.130 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.130 INFO:teuthology.orchestra.run.smithi190.stdout: # don't complain about insecure global_id in the test suite 2023-11-01T04:14:29.130 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-01T04:14:29.131 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-01T04:14:29.131 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.131 INFO:teuthology.orchestra.run.smithi190.stdout: # 1m isn't quite enough 2023-11-01T04:14:29.131 INFO:teuthology.orchestra.run.smithi190.stdout: mon_down_mkfs_grace = 2m 2023-11-01T04:14:29.131 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.131 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_filestore_osds = false 2023-11-01T04:14:29.132 INFO:teuthology.orchestra.run.smithi190.stdout: mon scrub interval = 300 2023-11-01T04:14:29.132 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-11-01T04:14:29.132 INFO:teuthology.orchestra.run.smithi190.stdout:[client] 2023-11-01T04:14:29.132 INFO:teuthology.orchestra.run.smithi190.stdout: rgw cache enabled = true 2023-11-01T04:14:29.132 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable ops log = true 2023-11-01T04:14:29.133 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable usage log = true 2023-11-01T04:14:29.133 INFO:teuthology.orchestra.run.smithi190.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-01T04:14:29.133 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-01T04:14:29.133 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.a] 2023-11-01T04:14:29.133 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.b] 2023-11-01T04:14:29.134 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.c] 2023-11-01T04:14:29.134 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2023-11-01T04:14:29.134 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2023-11-01T04:14:29.134 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-01T04:14:29.134 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2023-11-01T04:14:29.135 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.135 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2023-11-01T04:14:29.135 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.135 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2023-11-01T04:14:29.135 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.135 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2023-11-01T04:14:29.136 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2023-11-01T04:14:29.136 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.136 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2023-11-01T04:14:29.136 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2023-11-01T04:14:29.136 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.137 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-01T04:14:29.137 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2023-11-01T04:14:29.137 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.137 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2023-11-01T04:14:29.137 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-01T04:14:29.138 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.138 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2023-11-01T04:14:29.138 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.138 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2023-11-01T04:14:29.138 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2023-11-01T04:14:29.139 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2023-11-01T04:14:29.139 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2023-11-01T04:14:29.139 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2023-11-01T04:14:29.139 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2023-11-01T04:14:29.140 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2023-11-01T04:14:29.140 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-01T04:14:29.140 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-01T04:14:29.140 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2023-11-01T04:14:29.140 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.141 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-01T04:14:29.141 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.141 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2023-11-01T04:14:29.141 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.141 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2023-11-01T04:14:29.142 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.142 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2023-11-01T04:14:29.142 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2023-11-01T04:14:29.142 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2023-11-01T04:14:29.142 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.153,172.21.15.173,172.21.15.181" 2023-11-01T04:14:29.142 INFO:teuthology.orchestra.run.smithi181.stdout: mon election default strategy = 1 2023-11-01T04:14:29.143 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug inject dispatch delay duration = 0.1 2023-11-01T04:14:29.143 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug inject dispatch delay probability = 0.1 2023-11-01T04:14:29.143 INFO:teuthology.orchestra.run.smithi181.stdout: osd_object_clean_region_max_num_intervals = 1000 2023-11-01T04:14:29.143 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = 0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d 2023-11-01T04:14:29.143 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.144 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2023-11-01T04:14:29.144 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2023-11-01T04:14:29.144 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.144 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2023-11-01T04:14:29.144 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 120 2023-11-01T04:14:29.145 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2023-11-01T04:14:29.145 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.145 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2023-11-01T04:14:29.145 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2023-11-01T04:14:29.145 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.145 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2023-11-01T04:14:29.146 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2023-11-01T04:14:29.146 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2023-11-01T04:14:29.146 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2023-11-01T04:14:29.146 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.146 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2023-11-01T04:14:29.147 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2023-11-01T04:14:29.147 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.147 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2023-11-01T04:14:29.147 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.147 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2023-11-01T04:14:29.148 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.148 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2023-11-01T04:14:29.148 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.148 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2023-11-01T04:14:29.148 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2023-11-01T04:14:29.149 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.149 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2023-11-01T04:14:29.149 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2023-11-01T04:14:29.149 INFO:teuthology.orchestra.run.smithi181.stdout: debug monc = 20 2023-11-01T04:14:29.149 INFO:teuthology.orchestra.run.smithi181.stdout: bluefs allocator = hybrid 2023-11-01T04:14:29.150 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = hybrid 2023-11-01T04:14:29.150 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2023-11-01T04:14:29.150 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2023-11-01T04:14:29.150 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore zero block detection = True 2023-11-01T04:14:29.150 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 20 2023-11-01T04:14:29.151 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 20 2023-11-01T04:14:29.151 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2023-11-01T04:14:29.151 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2023-11-01T04:14:29.151 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 10 2023-11-01T04:14:29.151 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2023-11-01T04:14:29.152 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2023-11-01T04:14:29.152 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2023-11-01T04:14:29.152 INFO:teuthology.orchestra.run.smithi181.stdout: osd blocked scrub grace period = 3600 2023-11-01T04:14:29.152 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug reject backfill probability = 0.3 2023-11-01T04:14:29.152 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify cached snaps = True 2023-11-01T04:14:29.153 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify missing on start = True 2023-11-01T04:14:29.153 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2023-11-01T04:14:29.153 INFO:teuthology.orchestra.run.smithi181.stdout: osd max backfills = 3 2023-11-01T04:14:29.153 INFO:teuthology.orchestra.run.smithi181.stdout: osd max markdown count = 1000 2023-11-01T04:14:29.153 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock override recovery settings = True 2023-11-01T04:14:29.154 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2023-11-01T04:14:29.154 INFO:teuthology.orchestra.run.smithi181.stdout: osd op queue = debug_random 2023-11-01T04:14:29.154 INFO:teuthology.orchestra.run.smithi181.stdout: osd op queue cut off = debug_random 2023-11-01T04:14:29.154 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub min interval = 60 2023-11-01T04:14:29.154 INFO:teuthology.orchestra.run.smithi181.stdout: osd shutdown pgref assert = True 2023-11-01T04:14:29.155 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.155 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2023-11-01T04:14:29.155 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2023-11-01T04:14:29.155 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2023-11-01T04:14:29.155 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2023-11-01T04:14:29.155 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2023-11-01T04:14:29.156 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2023-11-01T04:14:29.156 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2023-11-01T04:14:29.156 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2023-11-01T04:14:29.156 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.157 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2023-11-01T04:14:29.157 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2023-11-01T04:14:29.157 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2023-11-01T04:14:29.157 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2023-11-01T04:14:29.157 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2023-11-01T04:14:29.158 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2023-11-01T04:14:29.158 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2023-11-01T04:14:29.158 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2023-11-01T04:14:29.158 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2023-11-01T04:14:29.158 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2023-11-01T04:14:29.158 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2023-11-01T04:14:29.159 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.159 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-01T04:14:29.159 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2023-11-01T04:14:29.159 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2023-11-01T04:14:29.160 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.160 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2023-11-01T04:14:29.160 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-01T04:14:29.160 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-01T04:14:29.160 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.161 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2023-11-01T04:14:29.161 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2023-11-01T04:14:29.161 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.161 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2023-11-01T04:14:29.161 INFO:teuthology.orchestra.run.smithi181.stdout: mon scrub interval = 300 2023-11-01T04:14:29.162 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-11-01T04:14:29.162 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2023-11-01T04:14:29.162 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2023-11-01T04:14:29.162 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2023-11-01T04:14:29.162 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2023-11-01T04:14:29.162 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-01T04:14:29.163 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-01T04:14:29.163 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2023-11-01T04:14:29.163 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2023-11-01T04:14:29.163 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2023-11-01T04:14:29.163 INFO:teuthology.orchestra.run.smithi173.stdout:[global] 2023-11-01T04:14:29.164 INFO:teuthology.orchestra.run.smithi173.stdout: chdir = "" 2023-11-01T04:14:29.164 INFO:teuthology.orchestra.run.smithi173.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-01T04:14:29.164 INFO:teuthology.orchestra.run.smithi173.stdout: auth supported = cephx 2023-11-01T04:14:29.164 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.165 INFO:teuthology.orchestra.run.smithi173.stdout: filestore xattr use omap = true 2023-11-01T04:14:29.165 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.165 INFO:teuthology.orchestra.run.smithi173.stdout: mon clock drift allowed = 1.000 2023-11-01T04:14:29.165 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.165 INFO:teuthology.orchestra.run.smithi173.stdout: osd crush chooseleaf type = 0 2023-11-01T04:14:29.166 INFO:teuthology.orchestra.run.smithi173.stdout: auth debug = true 2023-11-01T04:14:29.166 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.166 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on old message = true 2023-11-01T04:14:29.166 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on bug = true 2023-11-01T04:14:29.167 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.167 INFO:teuthology.orchestra.run.smithi173.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-01T04:14:29.167 INFO:teuthology.orchestra.run.smithi173.stdout: mon pg warn max object skew = 0 2023-11-01T04:14:29.167 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.167 INFO:teuthology.orchestra.run.smithi173.stdout: # disable pg_autoscaler by default for new pools 2023-11-01T04:14:29.168 INFO:teuthology.orchestra.run.smithi173.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-01T04:14:29.168 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.168 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default size = 2 2023-11-01T04:14:29.168 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.168 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow primary affinity = true 2023-11-01T04:14:29.169 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow pg remap = true 2023-11-01T04:14:29.169 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on legacy crush tunables = false 2023-11-01T04:14:29.169 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on crush straw calc version zero = false 2023-11-01T04:14:29.169 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on no sortbitwise = false 2023-11-01T04:14:29.169 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on osd down out interval zero = false 2023-11-01T04:14:29.170 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on too few osds = false 2023-11-01T04:14:29.170 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-01T04:14:29.170 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-01T04:14:29.170 INFO:teuthology.orchestra.run.smithi173.stdout: mon_allow_pool_size_one = true 2023-11-01T04:14:29.170 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.171 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-01T04:14:29.171 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.171 INFO:teuthology.orchestra.run.smithi173.stdout: osd default data pool replay window = 5 2023-11-01T04:14:29.171 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.171 INFO:teuthology.orchestra.run.smithi173.stdout: mon allow pool delete = true 2023-11-01T04:14:29.171 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.172 INFO:teuthology.orchestra.run.smithi173.stdout: mon cluster log file level = debug 2023-11-01T04:14:29.172 INFO:teuthology.orchestra.run.smithi173.stdout: debug asserts on shutdown = true 2023-11-01T04:14:29.172 INFO:teuthology.orchestra.run.smithi173.stdout: mon health detail to clog = false 2023-11-01T04:14:29.172 INFO:teuthology.orchestra.run.smithi173.stdout: mon host = "172.21.15.153,172.21.15.173,172.21.15.181" 2023-11-01T04:14:29.172 INFO:teuthology.orchestra.run.smithi173.stdout: mon election default strategy = 1 2023-11-01T04:14:29.173 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug inject dispatch delay duration = 0.1 2023-11-01T04:14:29.173 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug inject dispatch delay probability = 0.1 2023-11-01T04:14:29.173 INFO:teuthology.orchestra.run.smithi173.stdout: osd_object_clean_region_max_num_intervals = 1000 2023-11-01T04:14:29.173 INFO:teuthology.orchestra.run.smithi173.stdout: fsid = 0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d 2023-11-01T04:14:29.173 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.174 INFO:teuthology.orchestra.run.smithi173.stdout:[osd] 2023-11-01T04:14:29.174 INFO:teuthology.orchestra.run.smithi173.stdout: osd journal size = 100 2023-11-01T04:14:29.174 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.174 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub load threshold = 5.0 2023-11-01T04:14:29.175 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub max interval = 120 2023-11-01T04:14:29.175 INFO:teuthology.orchestra.run.smithi173.stdout: osd mclock profile = high_recovery_ops 2023-11-01T04:14:29.175 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.175 INFO:teuthology.orchestra.run.smithi173.stdout: osd recover clone overlap = true 2023-11-01T04:14:29.175 INFO:teuthology.orchestra.run.smithi173.stdout: osd recovery max chunk = 1048576 2023-11-01T04:14:29.175 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.176 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug shutdown = true 2023-11-01T04:14:29.176 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug op order = true 2023-11-01T04:14:29.176 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify stray on activate = true 2023-11-01T04:14:29.176 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug trim objects = true 2023-11-01T04:14:29.177 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.177 INFO:teuthology.orchestra.run.smithi173.stdout: osd open classes on start = true 2023-11-01T04:14:29.177 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug pg log writeout = true 2023-11-01T04:14:29.177 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.177 INFO:teuthology.orchestra.run.smithi173.stdout: osd deep scrub update digest min age = 30 2023-11-01T04:14:29.178 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.178 INFO:teuthology.orchestra.run.smithi173.stdout: osd map max advance = 10 2023-11-01T04:14:29.178 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.178 INFO:teuthology.orchestra.run.smithi173.stdout: journal zero on create = true 2023-11-01T04:14:29.179 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.179 INFO:teuthology.orchestra.run.smithi173.stdout: filestore ondisk finisher threads = 3 2023-11-01T04:14:29.179 INFO:teuthology.orchestra.run.smithi173.stdout: filestore apply finisher threads = 3 2023-11-01T04:14:29.179 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.180 INFO:teuthology.orchestra.run.smithi173.stdout: bdev debug aio = true 2023-11-01T04:14:29.180 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug misdirected ops = true 2023-11-01T04:14:29.180 INFO:teuthology.orchestra.run.smithi173.stdout: debug monc = 20 2023-11-01T04:14:29.180 INFO:teuthology.orchestra.run.smithi173.stdout: bluefs allocator = hybrid 2023-11-01T04:14:29.181 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore allocator = hybrid 2023-11-01T04:14:29.181 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore block size = 96636764160 2023-11-01T04:14:29.181 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore fsck on mount = True 2023-11-01T04:14:29.181 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore zero block detection = True 2023-11-01T04:14:29.181 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluefs = 20 2023-11-01T04:14:29.182 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluestore = 20 2023-11-01T04:14:29.182 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2023-11-01T04:14:29.182 INFO:teuthology.orchestra.run.smithi173.stdout: debug osd = 20 2023-11-01T04:14:29.182 INFO:teuthology.orchestra.run.smithi173.stdout: debug rocksdb = 10 2023-11-01T04:14:29.182 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd backfillfull_ratio = 0.85 2023-11-01T04:14:29.183 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd full ratio = 0.9 2023-11-01T04:14:29.183 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd nearfull ratio = 0.8 2023-11-01T04:14:29.183 INFO:teuthology.orchestra.run.smithi173.stdout: osd blocked scrub grace period = 3600 2023-11-01T04:14:29.183 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug reject backfill probability = 0.3 2023-11-01T04:14:29.183 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify cached snaps = True 2023-11-01T04:14:29.184 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify missing on start = True 2023-11-01T04:14:29.184 INFO:teuthology.orchestra.run.smithi173.stdout: osd failsafe full ratio = 0.95 2023-11-01T04:14:29.184 INFO:teuthology.orchestra.run.smithi173.stdout: osd max backfills = 3 2023-11-01T04:14:29.184 INFO:teuthology.orchestra.run.smithi173.stdout: osd max markdown count = 1000 2023-11-01T04:14:29.184 INFO:teuthology.orchestra.run.smithi173.stdout: osd mclock override recovery settings = True 2023-11-01T04:14:29.185 INFO:teuthology.orchestra.run.smithi173.stdout: osd objectstore = bluestore 2023-11-01T04:14:29.185 INFO:teuthology.orchestra.run.smithi173.stdout: osd op queue = debug_random 2023-11-01T04:14:29.185 INFO:teuthology.orchestra.run.smithi173.stdout: osd op queue cut off = debug_random 2023-11-01T04:14:29.185 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub min interval = 60 2023-11-01T04:14:29.185 INFO:teuthology.orchestra.run.smithi173.stdout: osd shutdown pgref assert = True 2023-11-01T04:14:29.186 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.186 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr] 2023-11-01T04:14:29.186 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2023-11-01T04:14:29.186 INFO:teuthology.orchestra.run.smithi173.stdout: debug mgr = 20 2023-11-01T04:14:29.186 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2023-11-01T04:14:29.187 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2023-11-01T04:14:29.187 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2023-11-01T04:14:29.187 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2023-11-01T04:14:29.187 INFO:teuthology.orchestra.run.smithi173.stdout: mgr/telemetry/nag = false 2023-11-01T04:14:29.187 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.188 INFO:teuthology.orchestra.run.smithi173.stdout:[mon] 2023-11-01T04:14:29.188 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2023-11-01T04:14:29.188 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2023-11-01T04:14:29.188 INFO:teuthology.orchestra.run.smithi173.stdout: debug paxos = 20 2023-11-01T04:14:29.188 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2023-11-01T04:14:29.188 INFO:teuthology.orchestra.run.smithi173.stdout: mon data avail warn = 5 2023-11-01T04:14:29.189 INFO:teuthology.orchestra.run.smithi173.stdout: mon mgr mkfs grace = 240 2023-11-01T04:14:29.189 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2023-11-01T04:14:29.189 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd reporter subtree level = osd 2023-11-01T04:14:29.189 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd prime pg temp = true 2023-11-01T04:14:29.189 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2023-11-01T04:14:29.190 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.190 INFO:teuthology.orchestra.run.smithi173.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-01T04:14:29.190 INFO:teuthology.orchestra.run.smithi173.stdout: auth mon ticket ttl = 660 # 11m 2023-11-01T04:14:29.190 INFO:teuthology.orchestra.run.smithi173.stdout: auth service ticket ttl = 240 # 4m 2023-11-01T04:14:29.190 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.191 INFO:teuthology.orchestra.run.smithi173.stdout: # don't complain about insecure global_id in the test suite 2023-11-01T04:14:29.191 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-01T04:14:29.191 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-01T04:14:29.191 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.191 INFO:teuthology.orchestra.run.smithi173.stdout: # 1m isn't quite enough 2023-11-01T04:14:29.192 INFO:teuthology.orchestra.run.smithi173.stdout: mon_down_mkfs_grace = 2m 2023-11-01T04:14:29.192 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.192 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_filestore_osds = false 2023-11-01T04:14:29.192 INFO:teuthology.orchestra.run.smithi173.stdout: mon scrub interval = 300 2023-11-01T04:14:29.192 INFO:teuthology.orchestra.run.smithi173.stdout: 2023-11-01T04:14:29.192 INFO:teuthology.orchestra.run.smithi173.stdout:[client] 2023-11-01T04:14:29.193 INFO:teuthology.orchestra.run.smithi173.stdout: rgw cache enabled = true 2023-11-01T04:14:29.193 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable ops log = true 2023-11-01T04:14:29.193 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable usage log = true 2023-11-01T04:14:29.193 INFO:teuthology.orchestra.run.smithi173.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-01T04:14:29.193 INFO:teuthology.orchestra.run.smithi173.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-01T04:14:29.194 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.a] 2023-11-01T04:14:29.194 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.b] 2023-11-01T04:14:29.194 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.c] 2023-11-01T04:14:29.194 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-01T04:14:29.195 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-01T04:14:29.244 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-01T04:14:29.245 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:14:29.245 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-01T04:14:29.250 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:14:29.250 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-01T04:14:29.295 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi153.front.sepia.ceph.com 2023-11-01T04:14:29.295 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:14:29.296 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-01T04:14:29.296 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-01T04:14:29.359 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:14:29.359 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:14:29.407 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi173.front.sepia.ceph.com 2023-11-01T04:14:29.408 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:14:29.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-01T04:14:29.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-01T04:14:29.430 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:14:29.430 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:14:29.477 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi181.front.sepia.ceph.com 2023-11-01T04:14:29.478 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:14:29.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-01T04:14:29.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-01T04:14:29.503 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:14:29.503 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:14:29.552 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi190.front.sepia.ceph.com 2023-11-01T04:14:29.552 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:14:29.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-01T04:14:29.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-01T04:14:29.575 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:14:29.575 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:14:29.623 INFO:tasks.ceph:Setting up mon nodes... 2023-11-01T04:14:29.623 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-01T04:14:29.624 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-11-01T04:14:29.667 INFO:teuthology.orchestra.run.smithi153.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-11-01T04:14:29.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-01T04:14:29.711 INFO:teuthology.orchestra.run.smithi190.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-01T04:14:29.731 INFO:tasks.ceph:Setting up mds nodes... 2023-11-01T04:14:29.732 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-01T04:14:29.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-01T04:14:29.811 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-01T04:14:29.821 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-01T04:14:29.822 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2023-11-01T04:14:29.822 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-01T04:14:29.835 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2023-11-01T04:14:29.835 INFO:tasks.ceph:role: osd.0 2023-11-01T04:14:29.835 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com 2023-11-01T04:14:29.836 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-01T04:14:29.899 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T04:14:29.900 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T04:14:29.900 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T04:14:29.900 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-01T04:14:29.900 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T04:14:29.900 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2023-11-01T04:14:29.901 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T04:14:29.901 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-01T04:14:29.901 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T04:14:29.901 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T04:14:30.362 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2023-11-01T04:14:30.364 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2023-11-01T04:14:30.365 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-01T04:14:30.457 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-01T04:14:30.468 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: /sbin/restorecon: command not found 2023-11-01T04:14:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:14:30.469 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-11-01T04:14:30.521 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2023-11-01T04:14:30.522 INFO:tasks.ceph:role: osd.4 2023-11-01T04:14:30.522 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com 2023-11-01T04:14:30.522 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-01T04:14:30.583 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T04:14:30.583 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T04:14:30.583 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T04:14:30.583 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-01T04:14:30.584 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T04:14:30.584 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2023-11-01T04:14:30.584 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T04:14:30.584 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-01T04:14:30.584 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T04:14:30.585 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T04:14:31.060 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2023-11-01T04:14:31.062 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2023-11-01T04:14:31.063 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2023-11-01T04:14:31.225 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-11-01T04:14:31.236 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: /sbin/restorecon: command not found 2023-11-01T04:14:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:14:31.237 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2023-11-01T04:14:31.291 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2023-11-01T04:14:31.292 INFO:tasks.ceph:role: osd.8 2023-11-01T04:14:31.292 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com 2023-11-01T04:14:31.292 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-01T04:14:31.355 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T04:14:31.355 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T04:14:31.356 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T04:14:31.356 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-01T04:14:31.356 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T04:14:31.356 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2023-11-01T04:14:31.356 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T04:14:31.356 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-01T04:14:31.357 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T04:14:31.357 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T04:14:31.836 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2023-11-01T04:14:31.839 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2023-11-01T04:14:31.839 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2023-11-01T04:14:31.984 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2023-11-01T04:14:31.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:14:31.996 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: /sbin/restorecon: command not found 2023-11-01T04:14:31.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2023-11-01T04:14:32.051 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2023-11-01T04:14:32.051 INFO:tasks.ceph:role: osd.12 2023-11-01T04:14:32.051 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi153.front.sepia.ceph.com 2023-11-01T04:14:32.052 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-01T04:14:32.115 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T04:14:32.115 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T04:14:32.115 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T04:14:32.116 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-01T04:14:32.116 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T04:14:32.116 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2023-11-01T04:14:32.116 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T04:14:32.116 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-01T04:14:32.117 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T04:14:32.117 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T04:14:32.581 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2023-11-01T04:14:32.583 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2023-11-01T04:14:32.584 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2023-11-01T04:14:32.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2023-11-01T04:14:32.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:14:32.694 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: /sbin/restorecon: command not found 2023-11-01T04:14:32.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:14:32.819 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:32.819+0000 7f5718a36640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-11-01T04:14:32.820 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:32.819+0000 7f5718a36640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-11-01T04:14:32.820 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:32.819+0000 7f5718a36640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-01T04:14:32.821 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:32.819+0000 7f5718a36640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-01T04:14:32.821 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:32.819+0000 7f5718a36640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-01T04:14:32.826 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:32.823+0000 7f5718a36640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-11-01T04:14:35.841 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-01T04:14:35.856 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:14:35.936 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:35.935+0000 7fe4e66ff640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-11-01T04:14:35.937 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:35.935+0000 7fe4e66ff640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-11-01T04:14:35.937 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:35.935+0000 7fe4e66ff640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-01T04:14:35.937 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:35.935+0000 7fe4e66ff640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-01T04:14:35.938 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:35.935+0000 7fe4e66ff640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-01T04:14:35.938 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:35.935+0000 7fe4e66ff640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-11-01T04:14:38.939 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-11-01T04:14:38.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:14:39.076 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:39.075+0000 7faef35cb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2023-11-01T04:14:39.077 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:39.075+0000 7faef35cb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2023-11-01T04:14:39.077 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:39.075+0000 7faef35cb640 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2023-11-01T04:14:39.077 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:39.075+0000 7faef35cb640 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2023-11-01T04:14:39.078 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:39.075+0000 7faef35cb640 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2023-11-01T04:14:39.078 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:39.075+0000 7faef35cb640 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2023-11-01T04:14:42.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2023-11-01T04:14:42.144 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:14:42.226 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:42.223+0000 7f862bce7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2023-11-01T04:14:42.227 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:42.227+0000 7f862bce7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2023-11-01T04:14:42.227 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:42.227+0000 7f862bce7640 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2023-11-01T04:14:42.227 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:42.227+0000 7f862bce7640 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2023-11-01T04:14:42.228 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:42.227+0000 7f862bce7640 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2023-11-01T04:14:42.228 INFO:teuthology.orchestra.run.smithi153.stderr:2023-11-01T04:14:42.227+0000 7f862bce7640 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2023-11-01T04:14:45.231 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2023-11-01T04:14:45.288 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-01T04:14:45.303 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2023-11-01T04:14:45.303 INFO:tasks.ceph:role: osd.1 2023-11-01T04:14:45.303 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi173.front.sepia.ceph.com 2023-11-01T04:14:45.304 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-01T04:14:45.365 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T04:14:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T04:14:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T04:14:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-01T04:14:45.366 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T04:14:45.366 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2023-11-01T04:14:45.366 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T04:14:45.366 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-01T04:14:45.367 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T04:14:45.367 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T04:14:45.814 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2023-11-01T04:14:45.816 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2023-11-01T04:14:45.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2023-11-01T04:14:45.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-01T04:14:45.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: /sbin/restorecon: command not found 2023-11-01T04:14:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:14:45.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-11-01T04:14:45.975 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2023-11-01T04:14:45.975 INFO:tasks.ceph:role: osd.5 2023-11-01T04:14:45.975 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi173.front.sepia.ceph.com 2023-11-01T04:14:45.976 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-01T04:14:46.038 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T04:14:46.038 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T04:14:46.038 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T04:14:46.038 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-01T04:14:46.039 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T04:14:46.039 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2023-11-01T04:14:46.039 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T04:14:46.039 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-01T04:14:46.039 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T04:14:46.040 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T04:14:46.505 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2023-11-01T04:14:46.506 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2023-11-01T04:14:46.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-11-01T04:14:46.668 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-11-01T04:14:46.680 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: /sbin/restorecon: command not found 2023-11-01T04:14:46.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:14:46.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2023-11-01T04:14:46.736 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2023-11-01T04:14:46.737 INFO:tasks.ceph:role: osd.9 2023-11-01T04:14:46.737 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi173.front.sepia.ceph.com 2023-11-01T04:14:46.737 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-01T04:14:46.803 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T04:14:46.803 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T04:14:46.803 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T04:14:46.804 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-01T04:14:46.804 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T04:14:46.804 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2023-11-01T04:14:46.804 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T04:14:46.804 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-01T04:14:46.804 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T04:14:46.805 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T04:14:47.275 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2023-11-01T04:14:47.277 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2023-11-01T04:14:47.277 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2023-11-01T04:14:47.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2023-11-01T04:14:47.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:14:47.424 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: /sbin/restorecon: command not found 2023-11-01T04:14:47.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2023-11-01T04:14:47.480 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2023-11-01T04:14:47.480 INFO:tasks.ceph:role: osd.13 2023-11-01T04:14:47.480 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi173.front.sepia.ceph.com 2023-11-01T04:14:47.481 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-01T04:14:47.539 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T04:14:47.540 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T04:14:47.540 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T04:14:47.540 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-01T04:14:47.540 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T04:14:47.540 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2023-11-01T04:14:47.540 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T04:14:47.541 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-01T04:14:47.541 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T04:14:47.541 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T04:14:47.965 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2023-11-01T04:14:47.968 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2023-11-01T04:14:47.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2023-11-01T04:14:48.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2023-11-01T04:14:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:14:48.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: /sbin/restorecon: command not found 2023-11-01T04:14:48.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:14:48.188 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:48.184+0000 7fbf31eb3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-11-01T04:14:48.191 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:48.188+0000 7fbf31eb3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-11-01T04:14:48.191 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:48.188+0000 7fbf31eb3640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-01T04:14:48.191 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:48.188+0000 7fbf31eb3640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-01T04:14:48.191 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:48.188+0000 7fbf31eb3640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-01T04:14:48.193 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:48.192+0000 7fbf31eb3640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-11-01T04:14:51.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-01T04:14:51.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:14:51.302 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:51.300+0000 7f2cca1ba640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-11-01T04:14:51.303 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:51.300+0000 7f2cca1ba640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-11-01T04:14:51.303 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:51.300+0000 7f2cca1ba640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-01T04:14:51.304 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:51.300+0000 7f2cca1ba640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-01T04:14:51.304 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:51.300+0000 7f2cca1ba640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-01T04:14:51.304 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:51.300+0000 7f2cca1ba640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-11-01T04:14:54.313 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-11-01T04:14:54.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:14:54.451 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:54.448+0000 7ff3519f8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2023-11-01T04:14:54.453 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:54.448+0000 7ff3519f8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2023-11-01T04:14:54.453 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:54.448+0000 7ff3519f8640 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2023-11-01T04:14:54.453 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:54.448+0000 7ff3519f8640 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2023-11-01T04:14:54.453 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:54.448+0000 7ff3519f8640 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2023-11-01T04:14:54.454 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:54.452+0000 7ff3519f8640 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2023-11-01T04:14:57.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2023-11-01T04:14:57.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:14:57.595 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:57.592+0000 7f5bb51c7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2023-11-01T04:14:57.596 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:57.592+0000 7f5bb51c7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2023-11-01T04:14:57.596 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:57.592+0000 7f5bb51c7640 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2023-11-01T04:14:57.596 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:57.592+0000 7f5bb51c7640 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2023-11-01T04:14:57.597 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:57.592+0000 7f5bb51c7640 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2023-11-01T04:14:57.597 INFO:teuthology.orchestra.run.smithi173.stderr:2023-11-01T04:14:57.596+0000 7f5bb51c7640 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2023-11-01T04:15:00.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2023-11-01T04:15:00.661 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-01T04:15:00.675 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2023-11-01T04:15:00.676 INFO:tasks.ceph:role: osd.2 2023-11-01T04:15:00.676 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com 2023-11-01T04:15:00.676 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-01T04:15:00.742 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T04:15:00.742 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T04:15:00.743 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T04:15:00.743 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-01T04:15:00.743 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T04:15:00.743 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2023-11-01T04:15:00.743 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T04:15:00.744 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-01T04:15:00.744 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T04:15:00.744 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T04:15:01.224 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2023-11-01T04:15:01.226 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2023-11-01T04:15:01.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2023-11-01T04:15:01.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-01T04:15:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:15:01.393 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: /sbin/restorecon: command not found 2023-11-01T04:15:01.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-11-01T04:15:01.447 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2023-11-01T04:15:01.447 INFO:tasks.ceph:role: osd.6 2023-11-01T04:15:01.447 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com 2023-11-01T04:15:01.448 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-01T04:15:01.510 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T04:15:01.510 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T04:15:01.511 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T04:15:01.511 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-01T04:15:01.511 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T04:15:01.511 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2023-11-01T04:15:01.512 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T04:15:01.512 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-01T04:15:01.512 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T04:15:01.512 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T04:15:01.960 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2023-11-01T04:15:01.962 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2023-11-01T04:15:01.962 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2023-11-01T04:15:02.044 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-11-01T04:15:02.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:15:02.096 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: /sbin/restorecon: command not found 2023-11-01T04:15:02.096 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2023-11-01T04:15:02.149 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2023-11-01T04:15:02.149 INFO:tasks.ceph:role: osd.10 2023-11-01T04:15:02.149 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com 2023-11-01T04:15:02.150 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-01T04:15:02.210 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T04:15:02.211 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T04:15:02.211 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T04:15:02.211 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-01T04:15:02.211 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T04:15:02.211 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2023-11-01T04:15:02.212 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T04:15:02.212 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-01T04:15:02.212 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T04:15:02.212 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T04:15:02.674 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2023-11-01T04:15:02.677 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2023-11-01T04:15:02.677 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2023-11-01T04:15:02.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2023-11-01T04:15:02.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:15:02.805 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: /sbin/restorecon: command not found 2023-11-01T04:15:02.805 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2023-11-01T04:15:02.859 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2023-11-01T04:15:02.859 INFO:tasks.ceph:role: osd.14 2023-11-01T04:15:02.859 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi181.front.sepia.ceph.com 2023-11-01T04:15:02.860 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-01T04:15:02.924 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T04:15:02.924 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T04:15:02.924 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T04:15:02.924 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-01T04:15:02.924 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T04:15:02.925 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2023-11-01T04:15:02.925 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T04:15:02.925 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-01T04:15:02.925 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T04:15:02.925 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T04:15:03.420 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2023-11-01T04:15:03.422 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2023-11-01T04:15:03.423 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2023-11-01T04:15:03.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2023-11-01T04:15:03.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:15:03.594 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: /sbin/restorecon: command not found 2023-11-01T04:15:03.595 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:15:03.812 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:03.810+0000 7f72b10ed640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-11-01T04:15:03.815 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:03.814+0000 7f72b10ed640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-11-01T04:15:03.816 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:03.814+0000 7f72b10ed640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-01T04:15:03.816 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:03.814+0000 7f72b10ed640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-01T04:15:03.816 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:03.814+0000 7f72b10ed640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-01T04:15:03.816 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:03.814+0000 7f72b10ed640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-11-01T04:15:06.827 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-01T04:15:06.841 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:15:06.923 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:06.922+0000 7f7251679640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-11-01T04:15:06.924 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:06.922+0000 7f7251679640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-11-01T04:15:06.924 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:06.922+0000 7f7251679640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-01T04:15:06.924 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:06.922+0000 7f7251679640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-01T04:15:06.924 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:06.922+0000 7f7251679640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-01T04:15:06.925 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:06.922+0000 7f7251679640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-11-01T04:15:09.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-11-01T04:15:09.991 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:15:10.074 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:10.070+0000 7fb6f4213640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2023-11-01T04:15:10.076 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:10.074+0000 7fb6f4213640 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2023-11-01T04:15:10.076 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:10.074+0000 7fb6f4213640 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2023-11-01T04:15:10.076 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:10.074+0000 7fb6f4213640 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2023-11-01T04:15:10.076 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:10.074+0000 7fb6f4213640 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2023-11-01T04:15:10.077 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:10.074+0000 7fb6f4213640 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2023-11-01T04:15:13.092 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2023-11-01T04:15:13.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:15:13.231 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:13.230+0000 7fe328796640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2023-11-01T04:15:13.232 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:13.230+0000 7fe328796640 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2023-11-01T04:15:13.232 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:13.230+0000 7fe328796640 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2023-11-01T04:15:13.233 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:13.230+0000 7fe328796640 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2023-11-01T04:15:13.233 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:13.230+0000 7fe328796640 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2023-11-01T04:15:13.234 INFO:teuthology.orchestra.run.smithi181.stderr:2023-11-01T04:15:13.230+0000 7fe328796640 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2023-11-01T04:15:16.238 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2023-11-01T04:15:16.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-11-01T04:15:16.307 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2023-11-01T04:15:16.307 INFO:tasks.ceph:role: osd.3 2023-11-01T04:15:16.307 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com 2023-11-01T04:15:16.308 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-01T04:15:16.370 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T04:15:16.370 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T04:15:16.371 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T04:15:16.371 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-01T04:15:16.371 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T04:15:16.371 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2023-11-01T04:15:16.371 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T04:15:16.372 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-01T04:15:16.372 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T04:15:16.372 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T04:15:16.824 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2023-11-01T04:15:16.827 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2023-11-01T04:15:16.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2023-11-01T04:15:16.927 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-11-01T04:15:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:15:16.937 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: /sbin/restorecon: command not found 2023-11-01T04:15:16.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-11-01T04:15:16.992 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2023-11-01T04:15:16.992 INFO:tasks.ceph:role: osd.7 2023-11-01T04:15:16.992 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com 2023-11-01T04:15:16.993 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-01T04:15:17.052 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T04:15:17.053 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T04:15:17.053 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T04:15:17.053 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-01T04:15:17.053 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T04:15:17.054 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2023-11-01T04:15:17.054 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T04:15:17.054 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-01T04:15:17.054 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T04:15:17.055 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T04:15:17.519 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2023-11-01T04:15:17.521 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2023-11-01T04:15:17.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2023-11-01T04:15:17.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-11-01T04:15:17.660 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: /sbin/restorecon: command not found 2023-11-01T04:15:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:15:17.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2023-11-01T04:15:17.718 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2023-11-01T04:15:17.718 INFO:tasks.ceph:role: osd.11 2023-11-01T04:15:17.719 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com 2023-11-01T04:15:17.719 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-01T04:15:17.782 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T04:15:17.782 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T04:15:17.782 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T04:15:17.782 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-01T04:15:17.783 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T04:15:17.783 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2023-11-01T04:15:17.783 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T04:15:17.783 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-01T04:15:17.784 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T04:15:17.784 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T04:15:18.271 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2023-11-01T04:15:18.273 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2023-11-01T04:15:18.274 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2023-11-01T04:15:18.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2023-11-01T04:15:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:15:18.439 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: /sbin/restorecon: command not found 2023-11-01T04:15:18.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2023-11-01T04:15:18.498 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2023-11-01T04:15:18.498 INFO:tasks.ceph:role: osd.15 2023-11-01T04:15:18.498 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi190.front.sepia.ceph.com 2023-11-01T04:15:18.499 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-01T04:15:18.561 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T04:15:18.562 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T04:15:18.562 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T04:15:18.562 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-01T04:15:18.563 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T04:15:18.563 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2023-11-01T04:15:18.563 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T04:15:18.563 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-01T04:15:18.564 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T04:15:18.564 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T04:15:19.011 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2023-11-01T04:15:19.013 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2023-11-01T04:15:19.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2023-11-01T04:15:19.099 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2023-11-01T04:15:19.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:15:19.152 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: /sbin/restorecon: command not found 2023-11-01T04:15:19.152 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:15:19.258 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:19.258+0000 7f4601d88640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-11-01T04:15:19.262 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:19.262+0000 7f4601d88640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-11-01T04:15:19.263 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:19.262+0000 7f4601d88640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-01T04:15:19.263 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:19.262+0000 7f4601d88640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-01T04:15:19.263 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:19.262+0000 7f4601d88640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-01T04:15:19.267 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:19.266+0000 7f4601d88640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-11-01T04:15:22.283 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-11-01T04:15:22.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:15:22.383 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:22.382+0000 7f709c8b8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-11-01T04:15:22.384 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:22.382+0000 7f709c8b8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-11-01T04:15:22.385 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:22.382+0000 7f709c8b8640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-01T04:15:22.385 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:22.382+0000 7f709c8b8640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-01T04:15:22.385 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:22.382+0000 7f709c8b8640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-01T04:15:22.385 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:22.382+0000 7f709c8b8640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-11-01T04:15:25.398 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-11-01T04:15:25.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:15:25.536 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:25.534+0000 7fce82c23640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2023-11-01T04:15:25.536 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:25.534+0000 7fce82c23640 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2023-11-01T04:15:25.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:25.534+0000 7fce82c23640 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2023-11-01T04:15:25.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:25.534+0000 7fce82c23640 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2023-11-01T04:15:25.537 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:25.534+0000 7fce82c23640 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2023-11-01T04:15:25.538 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:25.538+0000 7fce82c23640 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2023-11-01T04:15:28.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2023-11-01T04:15:28.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:15:28.676 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:28.674+0000 7fb225f61640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2023-11-01T04:15:28.677 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:28.674+0000 7fb225f61640 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2023-11-01T04:15:28.677 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:28.674+0000 7fb225f61640 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2023-11-01T04:15:28.677 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:28.674+0000 7fb225f61640 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2023-11-01T04:15:28.677 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:28.674+0000 7fb225f61640 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2023-11-01T04:15:28.678 INFO:teuthology.orchestra.run.smithi190.stderr:2023-11-01T04:15:28.678+0000 7fb225f61640 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2023-11-01T04:15:31.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2023-11-01T04:15:31.739 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-01T04:15:31.739 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:15:31.739 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-11-01T04:15:31.754 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:15:31.755 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-01T04:15:31.811 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:15:31.811 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-11-01T04:15:31.866 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:15:31.866 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2023-11-01T04:15:31.917 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:15:31.917 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2023-11-01T04:15:31.971 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:15:31.971 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-01T04:15:31.985 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:15:31.986 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-11-01T04:15:32.040 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:15:32.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2023-11-01T04:15:32.096 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:15:32.097 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2023-11-01T04:15:32.152 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:15:32.152 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-01T04:15:32.166 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:15:32.167 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-11-01T04:15:32.222 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:15:32.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2023-11-01T04:15:32.278 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:15:32.278 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2023-11-01T04:15:32.331 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:15:32.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-01T04:15:32.345 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:15:32.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-11-01T04:15:32.401 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:15:32.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-11-01T04:15:32.457 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:15:32.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2023-11-01T04:15:32.513 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:15:32.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2023-11-01T04:15:32.569 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:15:32.569 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-01T04:15:32.618 INFO:tasks.ceph:Adding keys to all mons... 2023-11-01T04:15:32.618 DEBUG:teuthology.orchestra.run.smithi153:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-01T04:15:32.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-01T04:15:32.623 DEBUG:teuthology.orchestra.run.smithi181:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-01T04:15:32.630 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.y] 2023-11-01T04:15:32.630 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCl0EFlRm3iJxAAcP2bWuZWkXPYuiY2nReMLA== 2023-11-01T04:15:32.631 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2023-11-01T04:15:32.631 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCo0EFl7Fn2MBAAEewkg8GexuwSwJYjWDluCA== 2023-11-01T04:15:32.631 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.4] 2023-11-01T04:15:32.631 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCr0EFlFb7tNxAAbYPEiKL3UWjKPqzdq/cqnQ== 2023-11-01T04:15:32.632 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.8] 2023-11-01T04:15:32.632 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCv0EFlnWmrBBAAVPRjydsiL329PSKDgJUdPw== 2023-11-01T04:15:32.632 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.12] 2023-11-01T04:15:32.632 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCy0EFlSFeeDRAAJ4tRqU9DHmiJGhdTv6uHUA== 2023-11-01T04:15:32.632 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2023-11-01T04:15:32.633 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQC40EFlhuRWCxAAshvmwYHKaAqZQuxMPj0OVw== 2023-11-01T04:15:32.633 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.5] 2023-11-01T04:15:32.633 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQC70EFlz5gpEhAAw3Ma1KY4saJkAP4WHF+i5A== 2023-11-01T04:15:32.633 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.9] 2023-11-01T04:15:32.634 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQC+0EFlwnsJGxAARjXKlxsB84mVWh443K13BQ== 2023-11-01T04:15:32.634 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.13] 2023-11-01T04:15:32.634 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDB0EFlThugIxAAQRBz5t0csbAMDneAPpSVlA== 2023-11-01T04:15:32.634 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2023-11-01T04:15:32.635 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDH0EFlz5SHMBAA9Ob0cme9mvFaX67KazbcSg== 2023-11-01T04:15:32.635 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.6] 2023-11-01T04:15:32.635 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDK0EFlgzcfNxAAanbo/6haIb+qkpHz2pW/iw== 2023-11-01T04:15:32.635 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.10] 2023-11-01T04:15:32.636 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDO0EFl6UaRBBAAv5mIj7iYslQfncHe1gdDvw== 2023-11-01T04:15:32.636 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.14] 2023-11-01T04:15:32.636 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDR0EFlUKLjDRAAAnotwDuDNE9iXCCRpZNm6Q== 2023-11-01T04:15:32.636 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.x] 2023-11-01T04:15:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCl0EFl08aaKhAABaitkQv53AVdEx+XATr71g== 2023-11-01T04:15:32.637 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.3] 2023-11-01T04:15:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDX0EFl3ymsDxAAZjo9Wu0hsrivSP996NWg3g== 2023-11-01T04:15:32.637 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.7] 2023-11-01T04:15:32.637 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDa0EFlZ2wiFxAAzttHDSqlekV2vxRAs2C/Aw== 2023-11-01T04:15:32.637 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.11] 2023-11-01T04:15:32.638 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDd0EFldxw2IBAAtl8p/etlaTVE6nTMGxxl0w== 2023-11-01T04:15:32.638 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.15] 2023-11-01T04:15:32.638 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDg0EFleSWRKBAAidSodRDZeKJbn5qaIH7++Q== 2023-11-01T04:15:32.638 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2023-11-01T04:15:32.638 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCl0EFl4mmHMBAA7AP8bp9K2PY/dxJliO9YuA== 2023-11-01T04:15:32.640 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr.y] 2023-11-01T04:15:32.641 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCl0EFlRm3iJxAAcP2bWuZWkXPYuiY2nReMLA== 2023-11-01T04:15:32.641 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.0] 2023-11-01T04:15:32.641 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCo0EFl7Fn2MBAAEewkg8GexuwSwJYjWDluCA== 2023-11-01T04:15:32.641 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.4] 2023-11-01T04:15:32.642 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCr0EFlFb7tNxAAbYPEiKL3UWjKPqzdq/cqnQ== 2023-11-01T04:15:32.642 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.8] 2023-11-01T04:15:32.642 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCv0EFlnWmrBBAAVPRjydsiL329PSKDgJUdPw== 2023-11-01T04:15:32.642 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.12] 2023-11-01T04:15:32.642 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCy0EFlSFeeDRAAJ4tRqU9DHmiJGhdTv6uHUA== 2023-11-01T04:15:32.643 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.1] 2023-11-01T04:15:32.643 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQC40EFlhuRWCxAAshvmwYHKaAqZQuxMPj0OVw== 2023-11-01T04:15:32.643 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.5] 2023-11-01T04:15:32.643 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQC70EFlz5gpEhAAw3Ma1KY4saJkAP4WHF+i5A== 2023-11-01T04:15:32.643 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.9] 2023-11-01T04:15:32.643 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQC+0EFlwnsJGxAARjXKlxsB84mVWh443K13BQ== 2023-11-01T04:15:32.644 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.13] 2023-11-01T04:15:32.644 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDB0EFlThugIxAAQRBz5t0csbAMDneAPpSVlA== 2023-11-01T04:15:32.644 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.2] 2023-11-01T04:15:32.644 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDH0EFlz5SHMBAA9Ob0cme9mvFaX67KazbcSg== 2023-11-01T04:15:32.644 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.6] 2023-11-01T04:15:32.645 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDK0EFlgzcfNxAAanbo/6haIb+qkpHz2pW/iw== 2023-11-01T04:15:32.645 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.10] 2023-11-01T04:15:32.645 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDO0EFl6UaRBBAAv5mIj7iYslQfncHe1gdDvw== 2023-11-01T04:15:32.645 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.14] 2023-11-01T04:15:32.645 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDR0EFlUKLjDRAAAnotwDuDNE9iXCCRpZNm6Q== 2023-11-01T04:15:32.646 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr.x] 2023-11-01T04:15:32.646 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCl0EFl08aaKhAABaitkQv53AVdEx+XATr71g== 2023-11-01T04:15:32.646 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.3] 2023-11-01T04:15:32.646 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDX0EFl3ymsDxAAZjo9Wu0hsrivSP996NWg3g== 2023-11-01T04:15:32.646 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.7] 2023-11-01T04:15:32.647 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDa0EFlZ2wiFxAAzttHDSqlekV2vxRAs2C/Aw== 2023-11-01T04:15:32.647 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.11] 2023-11-01T04:15:32.647 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDd0EFldxw2IBAAtl8p/etlaTVE6nTMGxxl0w== 2023-11-01T04:15:32.647 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.15] 2023-11-01T04:15:32.647 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQDg0EFleSWRKBAAidSodRDZeKJbn5qaIH7++Q== 2023-11-01T04:15:32.648 INFO:teuthology.orchestra.run.smithi181.stdout:[client.0] 2023-11-01T04:15:32.648 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCl0EFl4mmHMBAA7AP8bp9K2PY/dxJliO9YuA== 2023-11-01T04:15:32.648 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr.y] 2023-11-01T04:15:32.648 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCl0EFlRm3iJxAAcP2bWuZWkXPYuiY2nReMLA== 2023-11-01T04:15:32.649 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.0] 2023-11-01T04:15:32.649 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCo0EFl7Fn2MBAAEewkg8GexuwSwJYjWDluCA== 2023-11-01T04:15:32.649 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.4] 2023-11-01T04:15:32.649 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCr0EFlFb7tNxAAbYPEiKL3UWjKPqzdq/cqnQ== 2023-11-01T04:15:32.650 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.8] 2023-11-01T04:15:32.650 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCv0EFlnWmrBBAAVPRjydsiL329PSKDgJUdPw== 2023-11-01T04:15:32.650 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.12] 2023-11-01T04:15:32.650 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCy0EFlSFeeDRAAJ4tRqU9DHmiJGhdTv6uHUA== 2023-11-01T04:15:32.650 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.1] 2023-11-01T04:15:32.651 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQC40EFlhuRWCxAAshvmwYHKaAqZQuxMPj0OVw== 2023-11-01T04:15:32.651 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.5] 2023-11-01T04:15:32.651 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQC70EFlz5gpEhAAw3Ma1KY4saJkAP4WHF+i5A== 2023-11-01T04:15:32.651 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.9] 2023-11-01T04:15:32.652 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQC+0EFlwnsJGxAARjXKlxsB84mVWh443K13BQ== 2023-11-01T04:15:32.652 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.13] 2023-11-01T04:15:32.652 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQDB0EFlThugIxAAQRBz5t0csbAMDneAPpSVlA== 2023-11-01T04:15:32.652 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.2] 2023-11-01T04:15:32.652 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQDH0EFlz5SHMBAA9Ob0cme9mvFaX67KazbcSg== 2023-11-01T04:15:32.652 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.6] 2023-11-01T04:15:32.653 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQDK0EFlgzcfNxAAanbo/6haIb+qkpHz2pW/iw== 2023-11-01T04:15:32.653 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.10] 2023-11-01T04:15:32.653 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQDO0EFl6UaRBBAAv5mIj7iYslQfncHe1gdDvw== 2023-11-01T04:15:32.653 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.14] 2023-11-01T04:15:32.653 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQDR0EFlUKLjDRAAAnotwDuDNE9iXCCRpZNm6Q== 2023-11-01T04:15:32.654 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr.x] 2023-11-01T04:15:32.654 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCl0EFl08aaKhAABaitkQv53AVdEx+XATr71g== 2023-11-01T04:15:32.654 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.3] 2023-11-01T04:15:32.654 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQDX0EFl3ymsDxAAZjo9Wu0hsrivSP996NWg3g== 2023-11-01T04:15:32.654 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.7] 2023-11-01T04:15:32.655 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQDa0EFlZ2wiFxAAzttHDSqlekV2vxRAs2C/Aw== 2023-11-01T04:15:32.655 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.11] 2023-11-01T04:15:32.655 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQDd0EFldxw2IBAAtl8p/etlaTVE6nTMGxxl0w== 2023-11-01T04:15:32.655 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.15] 2023-11-01T04:15:32.655 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQDg0EFleSWRKBAAidSodRDZeKJbn5qaIH7++Q== 2023-11-01T04:15:32.655 INFO:teuthology.orchestra.run.smithi173.stdout:[client.0] 2023-11-01T04:15:32.656 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCl0EFl4mmHMBAA7AP8bp9K2PY/dxJliO9YuA== 2023-11-01T04:15:32.656 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-01T04:15:32.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-01T04:15:32.686 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-01T04:15:32.733 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:32.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:32.738 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:32.781 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:32.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:32.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:32.864 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:32.900 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:32.905 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:32.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:32.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:32.985 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.064 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.241 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.285 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.321 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.367 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.405 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.454 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.476 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.485 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.541 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.621 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.653 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.705 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-01T04:15:33.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-01T04:15:33.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-01T04:15:33.791 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.817 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.876 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.901 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.960 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:33.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:34.042 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:34.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:34.065 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T04:15:34.125 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-01T04:15:34.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-01T04:15:34.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-01T04:15:34.210 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-01T04:15:34.210 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-01T04:15:34.223 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-01T04:15:34.323 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-01T04:15:34.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-11-01T04:15:34.349 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-01T04:15:34.451 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-11-01T04:15:34.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-11-01T04:15:34.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-01T04:15:34.573 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-11-01T04:15:34.584 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:15:34.587 DEBUG:teuthology.orchestra.run.smithi173:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:15:34.590 DEBUG:teuthology.orchestra.run.smithi181:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-01T04:15:34.631 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-01T04:15:34.631 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-01T04:15:34.632 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-01T04:15:34.636 INFO:tasks.ceph.mon.a:Started 2023-11-01T04:15:34.636 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-01T04:15:34.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-01T04:15:34.639 INFO:tasks.ceph.mon.b:Started 2023-11-01T04:15:34.639 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-01T04:15:34.640 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-11-01T04:15:34.676 INFO:tasks.ceph.mon.c:Started 2023-11-01T04:15:34.676 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-01T04:15:34.677 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-11-01T04:15:34.677 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-11-01T04:15:34.680 INFO:tasks.ceph.mgr.y:Started 2023-11-01T04:15:34.680 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-01T04:15:34.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-01T04:15:34.683 INFO:tasks.ceph.mgr.x:Started 2023-11-01T04:15:34.684 INFO:tasks.ceph:Setting crush tunables to default 2023-11-01T04:15:34.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd crush tunables default 2023-11-01T04:15:45.079 INFO:teuthology.orchestra.run.smithi153.stderr:adjusted tunables profile to default 2023-11-01T04:15:45.098 INFO:tasks.ceph:check_enable_crimson: False 2023-11-01T04:15:45.099 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-01T04:15:45.099 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:15:45.100 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-01T04:15:45.112 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:15:45.112 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-11-01T04:15:45.169 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:15:45.170 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2023-11-01T04:15:45.171 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:45.170+0000 7fbd7c063180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-01T04:15:45.182 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2023-11-01T04:15:45.183 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2023-11-01T04:15:45.195 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:45.194+0000 7f06f70f5180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-01T04:15:45.242 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:15:45.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-01T04:15:45.258 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:15:45.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-11-01T04:15:45.312 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:15:45.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2023-11-01T04:15:45.372 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2023-11-01T04:15:45.372 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2023-11-01T04:15:45.428 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:15:45.428 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-01T04:15:45.444 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:15:45.444 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-11-01T04:15:45.498 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:15:45.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2023-11-01T04:15:45.551 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-11-01T04:15:45.552 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2023-11-01T04:15:45.606 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:15:45.607 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-11-01T04:15:45.621 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:15:45.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-11-01T04:15:45.676 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:15:45.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2023-11-01T04:15:45.715 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:45.714+0000 7fbd7c063180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-01T04:15:45.728 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-11-01T04:15:45.729 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2023-11-01T04:15:45.758 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:45.758+0000 7f06f70f5180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-01T04:15:45.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 83ce3480-defe-4c73-aa1d-47c7941a23ba 0 2023-11-01T04:15:45.887 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:45.886+0000 7fbd7c063180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-01T04:15:45.947 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:45.946+0000 7f06f70f5180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-01T04:15:46.021 INFO:teuthology.orchestra.run.smithi190.stdout:0 2023-11-01T04:15:46.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 01a9801f-72fb-4299-8708-d70eb59b0fb2 1 2023-11-01T04:15:46.235 INFO:teuthology.orchestra.run.smithi190.stdout:1 2023-11-01T04:15:46.251 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 7c16db19-9bec-4ca0-88eb-bb5fb673d160 2 2023-11-01T04:15:46.450 INFO:teuthology.orchestra.run.smithi190.stdout:2 2023-11-01T04:15:46.459 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:46.458+0000 7fbd7c063180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-01T04:15:46.466 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 4c4cc8d1-6761-4d20-90e8-7c3d43d889d8 3 2023-11-01T04:15:46.512 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:46.510+0000 7fbd7c063180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-01T04:15:46.525 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:46.526+0000 7f06f70f5180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-01T04:15:46.563 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:46.562+0000 7fbd7c063180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-01T04:15:46.580 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:46.578+0000 7f06f70f5180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-01T04:15:46.633 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:46.634+0000 7f06f70f5180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-01T04:15:46.640 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:46.638+0000 7fbd7c063180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-01T04:15:46.691 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:46.690+0000 7fbd7c063180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-01T04:15:46.710 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:46.710+0000 7f06f70f5180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-01T04:15:46.762 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:46.762+0000 7f06f70f5180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-01T04:15:46.763 INFO:teuthology.orchestra.run.smithi190.stdout:3 2023-11-01T04:15:46.768 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:46.766+0000 7fbd7c063180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-01T04:15:46.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 0fb14630-87e6-4456-9640-7986a5079821 4 2023-11-01T04:15:46.819 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:46.818+0000 7fbd7c063180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-01T04:15:46.842 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:46.842+0000 7f06f70f5180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-01T04:15:46.895 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:46.894+0000 7f06f70f5180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-01T04:15:46.959 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:46.958+0000 7fbd7c063180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-01T04:15:46.981 INFO:teuthology.orchestra.run.smithi190.stdout:4 2023-11-01T04:15:46.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new cc715599-0461-477b-beb2-8777d6f36b22 5 2023-11-01T04:15:47.019 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:47.018+0000 7fbd7c063180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-01T04:15:47.038 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:47.038+0000 7f06f70f5180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-01T04:15:47.098 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:47.098+0000 7f06f70f5180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-01T04:15:47.198 INFO:teuthology.orchestra.run.smithi190.stdout:5 2023-11-01T04:15:47.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new ca5bea1c-f309-4258-b85d-738097037962 6 2023-11-01T04:15:47.497 INFO:teuthology.orchestra.run.smithi190.stdout:6 2023-11-01T04:15:47.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 1e80e965-2a46-468c-bcb8-49bee8dd19c1 7 2023-11-01T04:15:47.681 INFO:tasks.ceph.mgr.y.smithi153.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-11-01T04:15:47.681 INFO:tasks.ceph.mgr.y.smithi153.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-11-01T04:15:47.682 INFO:tasks.ceph.mgr.y.smithi153.stderr: from numpy import show_config as show_numpy_config 2023-11-01T04:15:47.705 INFO:teuthology.orchestra.run.smithi190.stdout:7 2023-11-01T04:15:47.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 87179b74-818f-47df-ad70-1392cfe227e9 8 2023-11-01T04:15:47.724 INFO:tasks.ceph.mgr.x.smithi190.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-11-01T04:15:47.725 INFO:tasks.ceph.mgr.x.smithi190.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-11-01T04:15:47.725 INFO:tasks.ceph.mgr.x.smithi190.stderr: from numpy import show_config as show_numpy_config 2023-11-01T04:15:47.770 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:47.770+0000 7f06f70f5180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-01T04:15:47.777 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:47.774+0000 7fbd7c063180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-01T04:15:47.898 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:47.898+0000 7f06f70f5180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-01T04:15:47.930 INFO:teuthology.orchestra.run.smithi190.stdout:8 2023-11-01T04:15:47.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 10d3c443-1748-4a7c-8b60-b58cfd565b4d 9 2023-11-01T04:15:47.949 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:47.946+0000 7fbd7c063180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-01T04:15:48.059 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:48.058+0000 7f06f70f5180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-01T04:15:48.112 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:48.110+0000 7fbd7c063180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-01T04:15:48.124 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:48.122+0000 7f06f70f5180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-01T04:15:48.143 INFO:teuthology.orchestra.run.smithi190.stdout:9 2023-11-01T04:15:48.159 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 81bb6569-d02b-4638-8b76-f99756d415b5 10 2023-11-01T04:15:48.182 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:48.182+0000 7fbd7c063180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-01T04:15:48.251 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:48.250+0000 7f06f70f5180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-01T04:15:48.318 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:48.314+0000 7fbd7c063180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-01T04:15:48.351 INFO:teuthology.orchestra.run.smithi190.stdout:10 2023-11-01T04:15:48.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new b8aefe11-845f-48a4-930c-b05cdfbb460a 11 2023-11-01T04:15:48.555 INFO:teuthology.orchestra.run.smithi190.stdout:11 2023-11-01T04:15:48.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 2165437d-26ba-4e50-8646-8cedbc10da53 12 2023-11-01T04:15:48.773 INFO:teuthology.orchestra.run.smithi190.stdout:12 2023-11-01T04:15:48.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 7b9f38b4-f5ff-43f1-a373-2aaed710026d 13 2023-11-01T04:15:48.867 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:48.866+0000 7f06f70f5180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-01T04:15:48.937 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:48.934+0000 7fbd7c063180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-01T04:15:48.945 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:48.946+0000 7f06f70f5180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-01T04:15:48.991 INFO:teuthology.orchestra.run.smithi190.stdout:13 2023-11-01T04:15:49.008 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 054b27d2-8a4d-4851-8af8-78b71e6a537e 14 2023-11-01T04:15:49.013 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:49.010+0000 7fbd7c063180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-01T04:15:49.119 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:49.118+0000 7f06f70f5180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-01T04:15:49.183 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:49.182+0000 7fbd7c063180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-01T04:15:49.195 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:49.194+0000 7f06f70f5180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-01T04:15:49.210 INFO:teuthology.orchestra.run.smithi190.stdout:14 2023-11-01T04:15:49.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new e650f936-3712-47cd-9b17-d732991a27a3 15 2023-11-01T04:15:49.256 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:49.254+0000 7fbd7c063180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-01T04:15:49.424 INFO:teuthology.orchestra.run.smithi190.stdout:15 2023-11-01T04:15:49.441 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-01T04:15:49.441 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-11-01T04:15:49.444 INFO:tasks.ceph.osd.0:Started 2023-11-01T04:15:49.444 INFO:tasks.ceph.osd.4:Restarting daemon 2023-11-01T04:15:49.444 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-11-01T04:15:49.447 INFO:tasks.ceph.osd.4:Started 2023-11-01T04:15:49.448 INFO:tasks.ceph.osd.8:Restarting daemon 2023-11-01T04:15:49.448 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2023-11-01T04:15:49.450 INFO:tasks.ceph.osd.8:Started 2023-11-01T04:15:49.451 INFO:tasks.ceph.osd.12:Restarting daemon 2023-11-01T04:15:49.451 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2023-11-01T04:15:49.454 INFO:tasks.ceph.osd.12:Started 2023-11-01T04:15:49.454 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-01T04:15:49.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-11-01T04:15:49.457 INFO:tasks.ceph.osd.1:Started 2023-11-01T04:15:49.458 INFO:tasks.ceph.osd.5:Restarting daemon 2023-11-01T04:15:49.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-11-01T04:15:49.461 INFO:tasks.ceph.osd.5:Started 2023-11-01T04:15:49.461 INFO:tasks.ceph.osd.9:Restarting daemon 2023-11-01T04:15:49.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2023-11-01T04:15:49.464 INFO:tasks.ceph.osd.9:Started 2023-11-01T04:15:49.465 INFO:tasks.ceph.osd.13:Restarting daemon 2023-11-01T04:15:49.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2023-11-01T04:15:49.469 INFO:tasks.ceph.osd.13:Started 2023-11-01T04:15:49.470 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-01T04:15:49.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-11-01T04:15:49.473 INFO:tasks.ceph.osd.2:Started 2023-11-01T04:15:49.473 INFO:tasks.ceph.osd.6:Restarting daemon 2023-11-01T04:15:49.474 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-11-01T04:15:49.476 INFO:tasks.ceph.osd.6:Started 2023-11-01T04:15:49.477 INFO:tasks.ceph.osd.10:Restarting daemon 2023-11-01T04:15:49.477 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2023-11-01T04:15:49.480 INFO:tasks.ceph.osd.10:Started 2023-11-01T04:15:49.480 INFO:tasks.ceph.osd.14:Restarting daemon 2023-11-01T04:15:49.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2023-11-01T04:15:49.483 INFO:tasks.ceph.osd.14:Started 2023-11-01T04:15:49.483 INFO:tasks.ceph.osd.3:Restarting daemon 2023-11-01T04:15:49.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-11-01T04:15:49.486 INFO:tasks.ceph.osd.3:Started 2023-11-01T04:15:49.486 INFO:tasks.ceph.osd.7:Restarting daemon 2023-11-01T04:15:49.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-11-01T04:15:49.489 INFO:tasks.ceph.osd.7:Started 2023-11-01T04:15:49.489 INFO:tasks.ceph.osd.11:Restarting daemon 2023-11-01T04:15:49.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2023-11-01T04:15:49.490 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:49.486+0000 7f06f70f5180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-01T04:15:49.492 INFO:tasks.ceph.osd.11:Started 2023-11-01T04:15:49.493 INFO:tasks.ceph.osd.15:Restarting daemon 2023-11-01T04:15:49.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2023-11-01T04:15:49.495 INFO:tasks.ceph.osd.15:Started 2023-11-01T04:15:49.496 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-01T04:15:49.569 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:49.570+0000 7f06f70f5180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-01T04:15:49.578 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:49.574+0000 7fbd7c063180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-01T04:15:49.645 INFO:tasks.ceph.mgr.y.smithi153.stderr:2023-11-01T04:15:49.642+0000 7fbd7c063180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-01T04:15:49.684 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:15:49.685 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":18,"fsid":"0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d","created":"2023-11-01T04:15:44.902210+0000","modified":"2023-11-01T04:15:49.422762+0000","last_up_change":"0.000000","last_in_change":"2023-11-01T04:15:49.422762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"83ce3480-defe-4c73-aa1d-47c7941a23ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"01a9801f-72fb-4299-8708-d70eb59b0fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7c16db19-9bec-4ca0-88eb-bb5fb673d160","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4c4cc8d1-6761-4d20-90e8-7c3d43d889d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0fb14630-87e6-4456-9640-7986a5079821","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"cc715599-0461-477b-beb2-8777d6f36b22","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ca5bea1c-f309-4258-b85d-738097037962","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1e80e965-2a46-468c-bcb8-49bee8dd19c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"87179b74-818f-47df-ad70-1392cfe227e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"10d3c443-1748-4a7c-8b60-b58cfd565b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"81bb6569-d02b-4638-8b76-f99756d415b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"b8aefe11-845f-48a4-930c-b05cdfbb460a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"2165437d-26ba-4e50-8646-8cedbc10da53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"7b9f38b4-f5ff-43f1-a373-2aaed710026d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"054b27d2-8a4d-4851-8af8-78b71e6a537e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"e650f936-3712-47cd-9b17-d732991a27a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-01T04:15:49.701 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-01T04:15:49.702 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-01T04:15:50.064 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:15:50.062+0000 7f767d225640 -1 Falling back to public interface 2023-11-01T04:15:50.068 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:15:50.066+0000 7fb1c3b23640 -1 Falling back to public interface 2023-11-01T04:15:50.075 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:15:50.074+0000 7fa4864cc640 -1 Falling back to public interface 2023-11-01T04:15:50.083 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:15:50.082+0000 7fd30f881640 -1 Falling back to public interface 2023-11-01T04:15:50.088 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:15:50.085+0000 7f63a79c6640 -1 Falling back to public interface 2023-11-01T04:15:50.091 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:15:50.089+0000 7fc4f7362640 -1 Falling back to public interface 2023-11-01T04:15:50.098 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:15:50.096+0000 7fb292a52640 -1 Falling back to public interface 2023-11-01T04:15:50.098 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:15:50.098+0000 7fc0d08a9640 -1 Falling back to public interface 2023-11-01T04:15:50.103 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:15:50.101+0000 7f83d1004640 -1 Falling back to public interface 2023-11-01T04:15:50.104 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:15:50.101+0000 7f40559c6640 -1 Falling back to public interface 2023-11-01T04:15:50.105 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:15:50.104+0000 7f0bff89e640 -1 Falling back to public interface 2023-11-01T04:15:50.106 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:15:50.104+0000 7f07e45b6640 -1 Falling back to public interface 2023-11-01T04:15:50.106 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:15:50.106+0000 7f04e7ebd640 -1 Falling back to public interface 2023-11-01T04:15:50.107 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:15:50.106+0000 7f9be296f640 -1 Falling back to public interface 2023-11-01T04:15:50.114 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:15:50.112+0000 7efe095f4640 -1 Falling back to public interface 2023-11-01T04:15:50.118 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:15:50.118+0000 7f5c9a2b9640 -1 Falling back to public interface 2023-11-01T04:15:50.619 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:50.618+0000 7f069ea6a640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-01T04:15:50.619 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:15:50.618+0000 7f069ea6a640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-11-01T04:15:51.314 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-01T04:15:51.600 INFO:teuthology.misc.health.smithi153.stdout: 2023-11-01T04:15:51.600 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":18,"fsid":"0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d","created":"2023-11-01T04:15:44.902210+0000","modified":"2023-11-01T04:15:49.422762+0000","last_up_change":"0.000000","last_in_change":"2023-11-01T04:15:49.422762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"83ce3480-defe-4c73-aa1d-47c7941a23ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"01a9801f-72fb-4299-8708-d70eb59b0fb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7c16db19-9bec-4ca0-88eb-bb5fb673d160","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4c4cc8d1-6761-4d20-90e8-7c3d43d889d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0fb14630-87e6-4456-9640-7986a5079821","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"cc715599-0461-477b-beb2-8777d6f36b22","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ca5bea1c-f309-4258-b85d-738097037962","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1e80e965-2a46-468c-bcb8-49bee8dd19c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"87179b74-818f-47df-ad70-1392cfe227e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"10d3c443-1748-4a7c-8b60-b58cfd565b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"81bb6569-d02b-4638-8b76-f99756d415b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"b8aefe11-845f-48a4-930c-b05cdfbb460a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"2165437d-26ba-4e50-8646-8cedbc10da53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"7b9f38b4-f5ff-43f1-a373-2aaed710026d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"054b27d2-8a4d-4851-8af8-78b71e6a537e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"e650f936-3712-47cd-9b17-d732991a27a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-01T04:15:51.616 DEBUG:teuthology.misc:0 of 16 OSDs are up 2023-11-01T04:15:52.989 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:15:52.985+0000 7f40559c6640 -1 osd.6 0 log_to_monitors true 2023-11-01T04:15:53.661 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:15:53.658+0000 7fb1c3b23640 -1 osd.12 0 log_to_monitors true 2023-11-01T04:15:53.853 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:15:53.850+0000 7fa4864cc640 -1 osd.0 0 log_to_monitors true 2023-11-01T04:15:53.861 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:15:53.857+0000 7f04e7ebd640 -1 osd.11 0 log_to_monitors true 2023-11-01T04:15:53.873 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:15:53.870+0000 7f767d225640 -1 osd.8 0 log_to_monitors true 2023-11-01T04:15:53.889 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:15:53.889+0000 7f5c9a2b9640 -1 osd.3 0 log_to_monitors true 2023-11-01T04:15:53.915 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:15:53.911+0000 7fb292a52640 -1 osd.9 0 log_to_monitors true 2023-11-01T04:15:53.916 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:15:53.911+0000 7f07e45b6640 -1 osd.1 0 log_to_monitors true 2023-11-01T04:15:53.982 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:15:53.978+0000 7fd30f881640 -1 osd.4 0 log_to_monitors true 2023-11-01T04:15:54.014 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:15:54.011+0000 7efe095f4640 -1 osd.5 0 log_to_monitors true 2023-11-01T04:15:54.016 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:15:54.013+0000 7fc0d08a9640 -1 osd.7 0 log_to_monitors true 2023-11-01T04:15:54.019 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:15:54.015+0000 7f0bff89e640 -1 osd.13 0 log_to_monitors true 2023-11-01T04:15:54.033 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:15:54.029+0000 7fc4f7362640 -1 osd.14 0 log_to_monitors true 2023-11-01T04:15:54.036 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:15:54.033+0000 7f9be296f640 -1 osd.15 0 log_to_monitors true 2023-11-01T04:15:54.040 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:15:54.037+0000 7f63a79c6640 -1 osd.10 0 log_to_monitors true 2023-11-01T04:15:54.131 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:15:54.129+0000 7f83d1004640 -1 osd.2 0 log_to_monitors true 2023-11-01T04:15:54.633 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:15:54.629+0000 7f405196e640 -1 osd.6 0 waiting for initial osdmap 2023-11-01T04:15:54.663 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:15:54.661+0000 7f404afa6640 -1 osd.6 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-01T04:15:55.631 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:15:55.627+0000 7f0bfc05a640 -1 osd.13 0 waiting for initial osdmap 2023-11-01T04:15:55.631 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:15:55.627+0000 7f07e055e640 -1 osd.1 0 waiting for initial osdmap 2023-11-01T04:15:55.632 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:15:55.627+0000 7efe05db0640 -1 osd.5 0 waiting for initial osdmap 2023-11-01T04:15:55.633 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:15:55.629+0000 7f83ccfac640 -1 osd.2 0 waiting for initial osdmap 2023-11-01T04:15:55.633 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:15:55.629+0000 7f63a396e640 -1 osd.10 0 waiting for initial osdmap 2023-11-01T04:15:55.634 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:15:55.637+0000 7f9bdf12b640 -1 osd.15 0 waiting for initial osdmap 2023-11-01T04:15:55.644 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:15:55.645+0000 7f5c96261640 -1 osd.3 0 waiting for initial osdmap 2023-11-01T04:15:55.645 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:15:55.642+0000 7fa482474640 -1 osd.0 0 waiting for initial osdmap 2023-11-01T04:15:55.645 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:15:55.642+0000 7fb1c02df640 -1 osd.12 0 waiting for initial osdmap 2023-11-01T04:15:55.646 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:15:55.642+0000 7fd30b829640 -1 osd.4 0 waiting for initial osdmap 2023-11-01T04:15:55.646 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:15:55.642+0000 7f76791cd640 -1 osd.8 0 waiting for initial osdmap 2023-11-01T04:15:55.646 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:15:55.643+0000 7fb28e9fa640 -1 osd.9 0 waiting for initial osdmap 2023-11-01T04:15:55.647 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:15:55.645+0000 7f04e4679640 -1 osd.11 0 waiting for initial osdmap 2023-11-01T04:15:55.650 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:15:55.645+0000 7fc4f3b1e640 -1 osd.14 0 waiting for initial osdmap 2023-11-01T04:15:55.650 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:15:55.653+0000 7fc0cc851640 -1 osd.7 0 waiting for initial osdmap 2023-11-01T04:15:55.673 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:15:55.671+0000 7fb289835640 -1 osd.9 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-01T04:15:55.673 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:15:55.671+0000 7f0bf6e82640 -1 osd.13 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-01T04:15:55.674 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:15:55.671+0000 7efe00bd8640 -1 osd.5 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-01T04:15:55.683 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:15:55.681+0000 7f83c85e8640 -1 osd.2 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-01T04:15:55.683 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:15:55.681+0000 7f639efaa640 -1 osd.10 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-01T04:15:55.686 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:15:55.689+0000 7f04deca0640 -1 osd.11 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-01T04:15:55.687 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:15:55.689+0000 7f5c9189d640 -1 osd.3 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-01T04:15:55.687 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:15:55.689+0000 7f9bd9f53640 -1 osd.15 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-01T04:15:55.691 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:15:55.690+0000 7fb1ba105640 -1 osd.12 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-01T04:15:55.692 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:15:55.690+0000 7f7674008640 -1 osd.8 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-01T04:15:55.693 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:15:55.691+0000 7f07db399640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-01T04:15:55.695 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:15:55.693+0000 7fc4ee946640 -1 osd.14 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-01T04:15:55.707 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:15:55.706+0000 7fd306664640 -1 osd.4 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-01T04:15:55.710 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:15:55.713+0000 7fc0c7e8d640 -1 osd.7 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-01T04:15:55.715 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:15:55.714+0000 7fa47d2af640 -1 osd.0 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-01T04:15:59.228 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-01T04:15:59.530 INFO:teuthology.misc.health.smithi153.stdout: 2023-11-01T04:15:59.530 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":24,"fsid":"0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d","created":"2023-11-01T04:15:44.902210+0000","modified":"2023-11-01T04:15:58.694905+0000","last_up_change":"2023-11-01T04:15:56.634008+0000","last_in_change":"2023-11-01T04:15:49.422762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-01T04:15:57.685107+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83ce3480-defe-4c73-aa1d-47c7941a23ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6817","nonce":59702962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6819","nonce":59702962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6823","nonce":59702962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6821","nonce":59702962}]},"public_addr":"172.21.15.153:6817/59702962","cluster_addr":"172.21.15.153:6819/59702962","heartbeat_back_addr":"172.21.15.153:6823/59702962","heartbeat_front_addr":"172.21.15.153:6821/59702962","state":["exists","up"]},{"osd":1,"uuid":"01a9801f-72fb-4299-8708-d70eb59b0fb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6815","nonce":3103315277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6817","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6818","nonce":3103315277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6823","nonce":3103315277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6821","nonce":3103315277}]},"public_addr":"172.21.15.173:6815/3103315277","cluster_addr":"172.21.15.173:6818/3103315277","heartbeat_back_addr":"172.21.15.173:6823/3103315277","heartbeat_front_addr":"172.21.15.173:6821/3103315277","state":["exists","up"]},{"osd":2,"uuid":"7c16db19-9bec-4ca0-88eb-bb5fb673d160","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6817","nonce":1072240131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6820","nonce":1072240131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6825","nonce":1072240131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6821","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6823","nonce":1072240131}]},"public_addr":"172.21.15.181:6817/1072240131","cluster_addr":"172.21.15.181:6820/1072240131","heartbeat_back_addr":"172.21.15.181:6825/1072240131","heartbeat_front_addr":"172.21.15.181:6823/1072240131","state":["exists","up"]},{"osd":3,"uuid":"4c4cc8d1-6761-4d20-90e8-7c3d43d889d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6827","nonce":2304315422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6829","nonce":2304315422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6833","nonce":2304315422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6831","nonce":2304315422}]},"public_addr":"172.21.15.190:6827/2304315422","cluster_addr":"172.21.15.190:6829/2304315422","heartbeat_back_addr":"172.21.15.190:6833/2304315422","heartbeat_front_addr":"172.21.15.190:6831/2304315422","state":["exists","up"]},{"osd":4,"uuid":"0fb14630-87e6-4456-9640-7986a5079821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6825","nonce":3209586931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6827","nonce":3209586931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6831","nonce":3209586931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6829","nonce":3209586931}]},"public_addr":"172.21.15.153:6825/3209586931","cluster_addr":"172.21.15.153:6827/3209586931","heartbeat_back_addr":"172.21.15.153:6831/3209586931","heartbeat_front_addr":"172.21.15.153:6829/3209586931","state":["exists","up"]},{"osd":5,"uuid":"cc715599-0461-477b-beb2-8777d6f36b22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6825","nonce":409879231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6827","nonce":409879231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6831","nonce":409879231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6829","nonce":409879231}]},"public_addr":"172.21.15.173:6825/409879231","cluster_addr":"172.21.15.173:6827/409879231","heartbeat_back_addr":"172.21.15.173:6831/409879231","heartbeat_front_addr":"172.21.15.173:6829/409879231","state":["exists","up"]},{"osd":6,"uuid":"ca5bea1c-f309-4258-b85d-738097037962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6819","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6822","nonce":3465541770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6827","nonce":3465541770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6831","nonce":3465541770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6829","nonce":3465541770}]},"public_addr":"172.21.15.181:6822/3465541770","cluster_addr":"172.21.15.181:6827/3465541770","heartbeat_back_addr":"172.21.15.181:6831/3465541770","heartbeat_front_addr":"172.21.15.181:6829/3465541770","state":["exists","up"]},{"osd":7,"uuid":"1e80e965-2a46-468c-bcb8-49bee8dd19c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6803","nonce":1763196448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6805","nonce":1763196448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6809","nonce":1763196448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6807","nonce":1763196448}]},"public_addr":"172.21.15.190:6803/1763196448","cluster_addr":"172.21.15.190:6805/1763196448","heartbeat_back_addr":"172.21.15.190:6809/1763196448","heartbeat_front_addr":"172.21.15.190:6807/1763196448","state":["exists","up"]},{"osd":8,"uuid":"87179b74-818f-47df-ad70-1392cfe227e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6801","nonce":1068520389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6803","nonce":1068520389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6807","nonce":1068520389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6805","nonce":1068520389}]},"public_addr":"172.21.15.153:6801/1068520389","cluster_addr":"172.21.15.153:6803/1068520389","heartbeat_back_addr":"172.21.15.153:6807/1068520389","heartbeat_front_addr":"172.21.15.153:6805/1068520389","state":["exists","up"]},{"osd":9,"uuid":"10d3c443-1748-4a7c-8b60-b58cfd565b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6801","nonce":210565898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6803","nonce":210565898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6807","nonce":210565898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6805","nonce":210565898}]},"public_addr":"172.21.15.173:6801/210565898","cluster_addr":"172.21.15.173:6803/210565898","heartbeat_back_addr":"172.21.15.173:6807/210565898","heartbeat_front_addr":"172.21.15.173:6805/210565898","state":["exists","up"]},{"osd":10,"uuid":"81bb6569-d02b-4638-8b76-f99756d415b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6801","nonce":2007605188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6803","nonce":2007605188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6807","nonce":2007605188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6805","nonce":2007605188}]},"public_addr":"172.21.15.181:6801/2007605188","cluster_addr":"172.21.15.181:6803/2007605188","heartbeat_back_addr":"172.21.15.181:6807/2007605188","heartbeat_front_addr":"172.21.15.181:6805/2007605188","state":["exists","up"]},{"osd":11,"uuid":"b8aefe11-845f-48a4-930c-b05cdfbb460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6812","nonce":2456507663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6816","nonce":2456507663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6824","nonce":2456507663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6820","nonce":2456507663}]},"public_addr":"172.21.15.190:6812/2456507663","cluster_addr":"172.21.15.190:6816/2456507663","heartbeat_back_addr":"172.21.15.190:6824/2456507663","heartbeat_front_addr":"172.21.15.190:6820/2456507663","state":["exists","up"]},{"osd":12,"uuid":"2165437d-26ba-4e50-8646-8cedbc10da53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6809","nonce":819245857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6811","nonce":819245857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6815","nonce":819245857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6813","nonce":819245857}]},"public_addr":"172.21.15.153:6809/819245857","cluster_addr":"172.21.15.153:6811/819245857","heartbeat_back_addr":"172.21.15.153:6815/819245857","heartbeat_front_addr":"172.21.15.153:6813/819245857","state":["exists","up"]},{"osd":13,"uuid":"7b9f38b4-f5ff-43f1-a373-2aaed710026d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6809","nonce":3482699195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6811","nonce":3482699195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6819","nonce":3482699195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6814","nonce":3482699195}]},"public_addr":"172.21.15.173:6809/3482699195","cluster_addr":"172.21.15.173:6811/3482699195","heartbeat_back_addr":"172.21.15.173:6819/3482699195","heartbeat_front_addr":"172.21.15.173:6814/3482699195","state":["exists","up"]},{"osd":14,"uuid":"054b27d2-8a4d-4851-8af8-78b71e6a537e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6809","nonce":2780345165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6811","nonce":2780345165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6815","nonce":2780345165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6813","nonce":2780345165}]},"public_addr":"172.21.15.181:6809/2780345165","cluster_addr":"172.21.15.181:6811/2780345165","heartbeat_back_addr":"172.21.15.181:6815/2780345165","heartbeat_front_addr":"172.21.15.181:6813/2780345165","state":["exists","up"]},{"osd":15,"uuid":"e650f936-3712-47cd-9b17-d732991a27a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6813","nonce":3871145029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6815","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6817","nonce":3871145029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6823","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6825","nonce":3871145029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6819","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6821","nonce":3871145029}]},"public_addr":"172.21.15.190:6813/3871145029","cluster_addr":"172.21.15.190:6817/3871145029","heartbeat_back_addr":"172.21.15.190:6825/3871145029","heartbeat_front_addr":"172.21.15.190:6821/3871145029","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.898068+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.918555+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.123290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.920765+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.941559+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.003678+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.002527+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.028107+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.920344+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.893447+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.013907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.898025+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.650893+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.003356+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.008057+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.065472+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-01T04:15:59.548 DEBUG:teuthology.misc:16 of 16 OSDs are up 2023-11-01T04:15:59.548 INFO:tasks.ceph:Creating RBD pool 2023-11-01T04:15:59.549 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd pool create rbd 8 2023-11-01T04:16:00.738 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'rbd' created 2023-11-01T04:16:00.762 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-01T04:16:01.741 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-01T04:16:01.767 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-01T04:16:01.768 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-01T04:16:01.768 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-01T04:16:01.768 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-01T04:16:01.769 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-01T04:16:02.112 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:16:02.130 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3443540808},{"type":"v1","addr":"172.21.15.190:6801","nonce":3443540808}]},"active_addr":"172.21.15.190:6801/3443540808","active_change":"2023-11-01T04:15:49.592913+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":1674488683}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":700759198}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":3420815934}]}]} 2023-11-01T04:16:02.135 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-01T04:16:02.135 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-01T04:16:02.135 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-01T04:16:02.430 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:16:02.431 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":27,"fsid":"0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d","created":"2023-11-01T04:15:44.902210+0000","modified":"2023-11-01T04:16:01.739312+0000","last_up_change":"2023-11-01T04:15:56.634008+0000","last_in_change":"2023-11-01T04:15:49.422762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-01T04:15:57.685107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-01T04:15:59.882949+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":5.9899997711181641,"score_stable":5.9899997711181641,"optimal_score":0.68999999761581421,"raw_score_acting":4.130000114440918,"raw_score_stable":4.130000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83ce3480-defe-4c73-aa1d-47c7941a23ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6817","nonce":59702962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6819","nonce":59702962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6823","nonce":59702962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6821","nonce":59702962}]},"public_addr":"172.21.15.153:6817/59702962","cluster_addr":"172.21.15.153:6819/59702962","heartbeat_back_addr":"172.21.15.153:6823/59702962","heartbeat_front_addr":"172.21.15.153:6821/59702962","state":["exists","up"]},{"osd":1,"uuid":"01a9801f-72fb-4299-8708-d70eb59b0fb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6815","nonce":3103315277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6817","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6818","nonce":3103315277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6823","nonce":3103315277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6821","nonce":3103315277}]},"public_addr":"172.21.15.173:6815/3103315277","cluster_addr":"172.21.15.173:6818/3103315277","heartbeat_back_addr":"172.21.15.173:6823/3103315277","heartbeat_front_addr":"172.21.15.173:6821/3103315277","state":["exists","up"]},{"osd":2,"uuid":"7c16db19-9bec-4ca0-88eb-bb5fb673d160","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6817","nonce":1072240131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6820","nonce":1072240131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6825","nonce":1072240131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6821","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6823","nonce":1072240131}]},"public_addr":"172.21.15.181:6817/1072240131","cluster_addr":"172.21.15.181:6820/1072240131","heartbeat_back_addr":"172.21.15.181:6825/1072240131","heartbeat_front_addr":"172.21.15.181:6823/1072240131","state":["exists","up"]},{"osd":3,"uuid":"4c4cc8d1-6761-4d20-90e8-7c3d43d889d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6827","nonce":2304315422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6829","nonce":2304315422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6833","nonce":2304315422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6831","nonce":2304315422}]},"public_addr":"172.21.15.190:6827/2304315422","cluster_addr":"172.21.15.190:6829/2304315422","heartbeat_back_addr":"172.21.15.190:6833/2304315422","heartbeat_front_addr":"172.21.15.190:6831/2304315422","state":["exists","up"]},{"osd":4,"uuid":"0fb14630-87e6-4456-9640-7986a5079821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6825","nonce":3209586931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6827","nonce":3209586931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6831","nonce":3209586931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6829","nonce":3209586931}]},"public_addr":"172.21.15.153:6825/3209586931","cluster_addr":"172.21.15.153:6827/3209586931","heartbeat_back_addr":"172.21.15.153:6831/3209586931","heartbeat_front_addr":"172.21.15.153:6829/3209586931","state":["exists","up"]},{"osd":5,"uuid":"cc715599-0461-477b-beb2-8777d6f36b22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6825","nonce":409879231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6827","nonce":409879231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6831","nonce":409879231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6829","nonce":409879231}]},"public_addr":"172.21.15.173:6825/409879231","cluster_addr":"172.21.15.173:6827/409879231","heartbeat_back_addr":"172.21.15.173:6831/409879231","heartbeat_front_addr":"172.21.15.173:6829/409879231","state":["exists","up"]},{"osd":6,"uuid":"ca5bea1c-f309-4258-b85d-738097037962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6819","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6822","nonce":3465541770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6827","nonce":3465541770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6831","nonce":3465541770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6829","nonce":3465541770}]},"public_addr":"172.21.15.181:6822/3465541770","cluster_addr":"172.21.15.181:6827/3465541770","heartbeat_back_addr":"172.21.15.181:6831/3465541770","heartbeat_front_addr":"172.21.15.181:6829/3465541770","state":["exists","up"]},{"osd":7,"uuid":"1e80e965-2a46-468c-bcb8-49bee8dd19c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6803","nonce":1763196448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6805","nonce":1763196448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6809","nonce":1763196448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6807","nonce":1763196448}]},"public_addr":"172.21.15.190:6803/1763196448","cluster_addr":"172.21.15.190:6805/1763196448","heartbeat_back_addr":"172.21.15.190:6809/1763196448","heartbeat_front_addr":"172.21.15.190:6807/1763196448","state":["exists","up"]},{"osd":8,"uuid":"87179b74-818f-47df-ad70-1392cfe227e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6801","nonce":1068520389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6803","nonce":1068520389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6807","nonce":1068520389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6805","nonce":1068520389}]},"public_addr":"172.21.15.153:6801/1068520389","cluster_addr":"172.21.15.153:6803/1068520389","heartbeat_back_addr":"172.21.15.153:6807/1068520389","heartbeat_front_addr":"172.21.15.153:6805/1068520389","state":["exists","up"]},{"osd":9,"uuid":"10d3c443-1748-4a7c-8b60-b58cfd565b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6801","nonce":210565898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6803","nonce":210565898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6807","nonce":210565898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6805","nonce":210565898}]},"public_addr":"172.21.15.173:6801/210565898","cluster_addr":"172.21.15.173:6803/210565898","heartbeat_back_addr":"172.21.15.173:6807/210565898","heartbeat_front_addr":"172.21.15.173:6805/210565898","state":["exists","up"]},{"osd":10,"uuid":"81bb6569-d02b-4638-8b76-f99756d415b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6801","nonce":2007605188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6803","nonce":2007605188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6807","nonce":2007605188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6805","nonce":2007605188}]},"public_addr":"172.21.15.181:6801/2007605188","cluster_addr":"172.21.15.181:6803/2007605188","heartbeat_back_addr":"172.21.15.181:6807/2007605188","heartbeat_front_addr":"172.21.15.181:6805/2007605188","state":["exists","up"]},{"osd":11,"uuid":"b8aefe11-845f-48a4-930c-b05cdfbb460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6812","nonce":2456507663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6816","nonce":2456507663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6824","nonce":2456507663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6820","nonce":2456507663}]},"public_addr":"172.21.15.190:6812/2456507663","cluster_addr":"172.21.15.190:6816/2456507663","heartbeat_back_addr":"172.21.15.190:6824/2456507663","heartbeat_front_addr":"172.21.15.190:6820/2456507663","state":["exists","up"]},{"osd":12,"uuid":"2165437d-26ba-4e50-8646-8cedbc10da53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6809","nonce":819245857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6811","nonce":819245857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6815","nonce":819245857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6813","nonce":819245857}]},"public_addr":"172.21.15.153:6809/819245857","cluster_addr":"172.21.15.153:6811/819245857","heartbeat_back_addr":"172.21.15.153:6815/819245857","heartbeat_front_addr":"172.21.15.153:6813/819245857","state":["exists","up"]},{"osd":13,"uuid":"7b9f38b4-f5ff-43f1-a373-2aaed710026d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6809","nonce":3482699195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6811","nonce":3482699195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6819","nonce":3482699195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6814","nonce":3482699195}]},"public_addr":"172.21.15.173:6809/3482699195","cluster_addr":"172.21.15.173:6811/3482699195","heartbeat_back_addr":"172.21.15.173:6819/3482699195","heartbeat_front_addr":"172.21.15.173:6814/3482699195","state":["exists","up"]},{"osd":14,"uuid":"054b27d2-8a4d-4851-8af8-78b71e6a537e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6809","nonce":2780345165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6811","nonce":2780345165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6815","nonce":2780345165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6813","nonce":2780345165}]},"public_addr":"172.21.15.181:6809/2780345165","cluster_addr":"172.21.15.181:6811/2780345165","heartbeat_back_addr":"172.21.15.181:6815/2780345165","heartbeat_front_addr":"172.21.15.181:6813/2780345165","state":["exists","up"]},{"osd":15,"uuid":"e650f936-3712-47cd-9b17-d732991a27a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6813","nonce":3871145029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6815","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6817","nonce":3871145029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6823","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6825","nonce":3871145029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6819","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6821","nonce":3871145029}]},"public_addr":"172.21.15.190:6813/3871145029","cluster_addr":"172.21.15.190:6817/3871145029","heartbeat_back_addr":"172.21.15.190:6825/3871145029","heartbeat_front_addr":"172.21.15.190:6821/3871145029","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.898068+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.918555+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.123290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.920765+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.941559+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.003678+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.002527+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.028107+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.920344+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.893447+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.013907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.898025+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.650893+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.003356+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.008057+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.065472+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-01T04:16:02.449 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-01T04:16:02.449 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-01T04:16:02.734 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:16:02.735 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":27,"fsid":"0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d","created":"2023-11-01T04:15:44.902210+0000","modified":"2023-11-01T04:16:01.739312+0000","last_up_change":"2023-11-01T04:15:56.634008+0000","last_in_change":"2023-11-01T04:15:49.422762+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-01T04:15:57.685107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-01T04:15:59.882949+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":5.9899997711181641,"score_stable":5.9899997711181641,"optimal_score":0.68999999761581421,"raw_score_acting":4.130000114440918,"raw_score_stable":4.130000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83ce3480-defe-4c73-aa1d-47c7941a23ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6817","nonce":59702962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6819","nonce":59702962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6823","nonce":59702962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6821","nonce":59702962}]},"public_addr":"172.21.15.153:6817/59702962","cluster_addr":"172.21.15.153:6819/59702962","heartbeat_back_addr":"172.21.15.153:6823/59702962","heartbeat_front_addr":"172.21.15.153:6821/59702962","state":["exists","up"]},{"osd":1,"uuid":"01a9801f-72fb-4299-8708-d70eb59b0fb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6815","nonce":3103315277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6817","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6818","nonce":3103315277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6823","nonce":3103315277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6821","nonce":3103315277}]},"public_addr":"172.21.15.173:6815/3103315277","cluster_addr":"172.21.15.173:6818/3103315277","heartbeat_back_addr":"172.21.15.173:6823/3103315277","heartbeat_front_addr":"172.21.15.173:6821/3103315277","state":["exists","up"]},{"osd":2,"uuid":"7c16db19-9bec-4ca0-88eb-bb5fb673d160","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6817","nonce":1072240131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6820","nonce":1072240131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6825","nonce":1072240131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6821","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6823","nonce":1072240131}]},"public_addr":"172.21.15.181:6817/1072240131","cluster_addr":"172.21.15.181:6820/1072240131","heartbeat_back_addr":"172.21.15.181:6825/1072240131","heartbeat_front_addr":"172.21.15.181:6823/1072240131","state":["exists","up"]},{"osd":3,"uuid":"4c4cc8d1-6761-4d20-90e8-7c3d43d889d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6827","nonce":2304315422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6829","nonce":2304315422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6833","nonce":2304315422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6831","nonce":2304315422}]},"public_addr":"172.21.15.190:6827/2304315422","cluster_addr":"172.21.15.190:6829/2304315422","heartbeat_back_addr":"172.21.15.190:6833/2304315422","heartbeat_front_addr":"172.21.15.190:6831/2304315422","state":["exists","up"]},{"osd":4,"uuid":"0fb14630-87e6-4456-9640-7986a5079821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6825","nonce":3209586931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6827","nonce":3209586931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6831","nonce":3209586931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6829","nonce":3209586931}]},"public_addr":"172.21.15.153:6825/3209586931","cluster_addr":"172.21.15.153:6827/3209586931","heartbeat_back_addr":"172.21.15.153:6831/3209586931","heartbeat_front_addr":"172.21.15.153:6829/3209586931","state":["exists","up"]},{"osd":5,"uuid":"cc715599-0461-477b-beb2-8777d6f36b22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6825","nonce":409879231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6827","nonce":409879231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6831","nonce":409879231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6829","nonce":409879231}]},"public_addr":"172.21.15.173:6825/409879231","cluster_addr":"172.21.15.173:6827/409879231","heartbeat_back_addr":"172.21.15.173:6831/409879231","heartbeat_front_addr":"172.21.15.173:6829/409879231","state":["exists","up"]},{"osd":6,"uuid":"ca5bea1c-f309-4258-b85d-738097037962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6819","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6822","nonce":3465541770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6827","nonce":3465541770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6831","nonce":3465541770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6829","nonce":3465541770}]},"public_addr":"172.21.15.181:6822/3465541770","cluster_addr":"172.21.15.181:6827/3465541770","heartbeat_back_addr":"172.21.15.181:6831/3465541770","heartbeat_front_addr":"172.21.15.181:6829/3465541770","state":["exists","up"]},{"osd":7,"uuid":"1e80e965-2a46-468c-bcb8-49bee8dd19c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6803","nonce":1763196448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6805","nonce":1763196448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6809","nonce":1763196448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6807","nonce":1763196448}]},"public_addr":"172.21.15.190:6803/1763196448","cluster_addr":"172.21.15.190:6805/1763196448","heartbeat_back_addr":"172.21.15.190:6809/1763196448","heartbeat_front_addr":"172.21.15.190:6807/1763196448","state":["exists","up"]},{"osd":8,"uuid":"87179b74-818f-47df-ad70-1392cfe227e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6801","nonce":1068520389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6803","nonce":1068520389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6807","nonce":1068520389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6805","nonce":1068520389}]},"public_addr":"172.21.15.153:6801/1068520389","cluster_addr":"172.21.15.153:6803/1068520389","heartbeat_back_addr":"172.21.15.153:6807/1068520389","heartbeat_front_addr":"172.21.15.153:6805/1068520389","state":["exists","up"]},{"osd":9,"uuid":"10d3c443-1748-4a7c-8b60-b58cfd565b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6801","nonce":210565898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6803","nonce":210565898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6807","nonce":210565898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6805","nonce":210565898}]},"public_addr":"172.21.15.173:6801/210565898","cluster_addr":"172.21.15.173:6803/210565898","heartbeat_back_addr":"172.21.15.173:6807/210565898","heartbeat_front_addr":"172.21.15.173:6805/210565898","state":["exists","up"]},{"osd":10,"uuid":"81bb6569-d02b-4638-8b76-f99756d415b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6801","nonce":2007605188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6803","nonce":2007605188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6807","nonce":2007605188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6805","nonce":2007605188}]},"public_addr":"172.21.15.181:6801/2007605188","cluster_addr":"172.21.15.181:6803/2007605188","heartbeat_back_addr":"172.21.15.181:6807/2007605188","heartbeat_front_addr":"172.21.15.181:6805/2007605188","state":["exists","up"]},{"osd":11,"uuid":"b8aefe11-845f-48a4-930c-b05cdfbb460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6812","nonce":2456507663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6816","nonce":2456507663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6824","nonce":2456507663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6820","nonce":2456507663}]},"public_addr":"172.21.15.190:6812/2456507663","cluster_addr":"172.21.15.190:6816/2456507663","heartbeat_back_addr":"172.21.15.190:6824/2456507663","heartbeat_front_addr":"172.21.15.190:6820/2456507663","state":["exists","up"]},{"osd":12,"uuid":"2165437d-26ba-4e50-8646-8cedbc10da53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6809","nonce":819245857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6811","nonce":819245857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6815","nonce":819245857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6813","nonce":819245857}]},"public_addr":"172.21.15.153:6809/819245857","cluster_addr":"172.21.15.153:6811/819245857","heartbeat_back_addr":"172.21.15.153:6815/819245857","heartbeat_front_addr":"172.21.15.153:6813/819245857","state":["exists","up"]},{"osd":13,"uuid":"7b9f38b4-f5ff-43f1-a373-2aaed710026d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6809","nonce":3482699195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6811","nonce":3482699195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6819","nonce":3482699195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6814","nonce":3482699195}]},"public_addr":"172.21.15.173:6809/3482699195","cluster_addr":"172.21.15.173:6811/3482699195","heartbeat_back_addr":"172.21.15.173:6819/3482699195","heartbeat_front_addr":"172.21.15.173:6814/3482699195","state":["exists","up"]},{"osd":14,"uuid":"054b27d2-8a4d-4851-8af8-78b71e6a537e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6809","nonce":2780345165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6811","nonce":2780345165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6815","nonce":2780345165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6813","nonce":2780345165}]},"public_addr":"172.21.15.181:6809/2780345165","cluster_addr":"172.21.15.181:6811/2780345165","heartbeat_back_addr":"172.21.15.181:6815/2780345165","heartbeat_front_addr":"172.21.15.181:6813/2780345165","state":["exists","up"]},{"osd":15,"uuid":"e650f936-3712-47cd-9b17-d732991a27a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6813","nonce":3871145029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6815","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6817","nonce":3871145029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6823","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6825","nonce":3871145029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6819","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6821","nonce":3871145029}]},"public_addr":"172.21.15.190:6813/3871145029","cluster_addr":"172.21.15.190:6817/3871145029","heartbeat_back_addr":"172.21.15.190:6825/3871145029","heartbeat_front_addr":"172.21.15.190:6821/3871145029","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.898068+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.918555+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.123290+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.920765+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.941559+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.003678+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.002527+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.028107+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.920344+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.893447+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.013907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.898025+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.650893+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.003356+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.008057+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.065472+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-01T04:16:02.752 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-01T04:16:02.753 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-01T04:16:02.753 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-01T04:16:02.754 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-11-01T04:16:02.754 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-11-01T04:16:02.755 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-11-01T04:16:02.756 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-11-01T04:16:02.756 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-11-01T04:16:02.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2023-11-01T04:16:02.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2023-11-01T04:16:02.758 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2023-11-01T04:16:02.759 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2023-11-01T04:16:02.759 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2023-11-01T04:16:02.760 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2023-11-01T04:16:02.760 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2023-11-01T04:16:02.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2023-11-01T04:16:03.213 INFO:teuthology.orchestra.run.smithi153.stdout:94489280517 2023-11-01T04:16:03.213 INFO:teuthology.orchestra.run.smithi153.stdout:94489280515 2023-11-01T04:16:03.214 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2023-11-01T04:16:03.215 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-01T04:16:03.222 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:03.222 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-01T04:16:03.231 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:03.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-01T04:16:03.261 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:03.261 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-01T04:16:03.278 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:03.279 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2023-11-01T04:16:03.289 INFO:teuthology.orchestra.run.smithi153.stdout:90194313221 2023-11-01T04:16:03.290 INFO:teuthology.orchestra.run.smithi153.stdout:94489280517 2023-11-01T04:16:03.290 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2023-11-01T04:16:03.291 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-01T04:16:03.292 INFO:teuthology.orchestra.run.smithi153.stdout:94489280517 2023-11-01T04:16:03.292 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2023-11-01T04:16:03.298 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:03.298 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-01T04:16:03.300 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:03.301 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-01T04:16:03.303 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:03.304 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2023-11-01T04:16:03.307 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:03.307 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2023-11-01T04:16:03.314 INFO:teuthology.orchestra.run.smithi153.stdout:94489280515 2023-11-01T04:16:03.315 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2023-11-01T04:16:03.320 INFO:teuthology.orchestra.run.smithi153.stdout:94489280515 2023-11-01T04:16:03.321 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-01T04:16:03.336 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:03.336 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2023-11-01T04:16:04.042 INFO:teuthology.orchestra.run.smithi153.stdout:94489280517 2023-11-01T04:16:04.063 INFO:teuthology.orchestra.run.smithi153.stdout:94489280515 2023-11-01T04:16:04.115 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.8 2023-11-01T04:16:04.115 DEBUG:teuthology.parallel:result is None 2023-11-01T04:16:04.119 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:04.125 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.11 2023-11-01T04:16:04.126 DEBUG:teuthology.parallel:result is None 2023-11-01T04:16:04.137 INFO:teuthology.orchestra.run.smithi153.stdout:94489280517 2023-11-01T04:16:04.181 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.2 2023-11-01T04:16:04.181 DEBUG:teuthology.parallel:result is None 2023-11-01T04:16:04.183 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.3 2023-11-01T04:16:04.183 DEBUG:teuthology.parallel:result is None 2023-11-01T04:16:04.236 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:04.236 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:04.238 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:04.263 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.9 2023-11-01T04:16:04.263 DEBUG:teuthology.parallel:result is None 2023-11-01T04:16:04.281 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.1 2023-11-01T04:16:04.282 DEBUG:teuthology.parallel:result is None 2023-11-01T04:16:04.285 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.4 2023-11-01T04:16:04.286 DEBUG:teuthology.parallel:result is None 2023-11-01T04:16:04.307 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:04.331 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:04.331 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:04.334 INFO:teuthology.orchestra.run.smithi153.stdout:90194313221 2023-11-01T04:16:04.357 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:04.358 INFO:teuthology.orchestra.run.smithi153.stdout:94489280516 2023-11-01T04:16:04.359 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.10 2023-11-01T04:16:04.359 DEBUG:teuthology.parallel:result is None 2023-11-01T04:16:04.365 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.0 2023-11-01T04:16:04.366 DEBUG:teuthology.parallel:result is None 2023-11-01T04:16:04.368 INFO:teuthology.orchestra.run.smithi153.stdout:94489280515 2023-11-01T04:16:04.375 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.13 2023-11-01T04:16:04.375 DEBUG:teuthology.parallel:result is None 2023-11-01T04:16:04.376 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313221 got 90194313221 for osd.6 2023-11-01T04:16:04.376 INFO:teuthology.orchestra.run.smithi153.stdout:94489280517 2023-11-01T04:16:04.377 DEBUG:teuthology.parallel:result is None 2023-11-01T04:16:04.381 INFO:teuthology.orchestra.run.smithi153.stdout:94489280515 2023-11-01T04:16:04.389 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.15 2023-11-01T04:16:04.390 DEBUG:teuthology.parallel:result is None 2023-11-01T04:16:04.396 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.5 2023-11-01T04:16:04.396 DEBUG:teuthology.parallel:result is None 2023-11-01T04:16:04.401 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.12 2023-11-01T04:16:04.402 DEBUG:teuthology.parallel:result is None 2023-11-01T04:16:04.404 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.14 2023-11-01T04:16:04.404 DEBUG:teuthology.parallel:result is None 2023-11-01T04:16:04.405 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.7 2023-11-01T04:16:04.405 DEBUG:teuthology.parallel:result is None 2023-11-01T04:16:04.405 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-01T04:16:04.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-01T04:16:04.713 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:16:04.714 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-11-01T04:16:04.730 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-11-01T04:16:03.604251+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":437612,"kb_used_data":5064,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509511828,"statfs":{"total":1546188226560,"available":1545740111872,"internally_reserved":0,"allocated":5185536,"data_stored":2449650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25505,"internal_metadata":442473567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":131072,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.837382"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:01.955150+0000","last_change":"2023-11-01T04:16:01.855207+0000","last_active":"2023-11-01T04:16:01.955150+0000","last_peered":"2023-11-01T04:16:01.955150+0000","last_clean":"2023-11-01T04:16:01.955150+0000","last_became_active":"2023-11-01T04:16:01.854811+0000","last_became_peered":"2023-11-01T04:16:01.854811+0000","last_unstale":"2023-11-01T04:16:01.955150+0000","last_undegraded":"2023-11-01T04:16:01.955150+0000","last_fullsized":"2023-11-01T04:16:01.955150+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:07.050216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-11-01T04:16:03.146049+0000","last_change":"2023-11-01T04:16:01.875964+0000","last_active":"2023-11-01T04:16:03.146049+0000","last_peered":"2023-11-01T04:16:03.146049+0000","last_clean":"2023-11-01T04:16:03.146049+0000","last_became_active":"2023-11-01T04:16:01.875255+0000","last_became_peered":"2023-11-01T04:16:01.875255+0000","last_unstale":"2023-11-01T04:16:03.146049+0000","last_undegraded":"2023-11-01T04:16:03.146049+0000","last_fullsized":"2023-11-01T04:16:03.146049+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:05.611210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11],"acting":[14,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:02.169746+0000","last_change":"2023-11-01T04:16:02.168355+0000","last_active":"2023-11-01T04:16:02.169746+0000","last_peered":"2023-11-01T04:16:02.169746+0000","last_clean":"2023-11-01T04:16:02.169746+0000","last_became_active":"2023-11-01T04:16:02.167473+0000","last_became_peered":"2023-11-01T04:16:02.167473+0000","last_unstale":"2023-11-01T04:16:02.169746+0000","last_undegraded":"2023-11-01T04:16:02.169746+0000","last_fullsized":"2023-11-01T04:16:02.169746+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:03.374641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:01.756078+0000","last_change":"2023-11-01T04:16:01.756078+0000","last_active":"2023-11-01T04:16:01.756078+0000","last_peered":"2023-11-01T04:16:01.756078+0000","last_clean":"2023-11-01T04:16:01.756078+0000","last_became_active":"2023-11-01T04:16:01.755533+0000","last_became_peered":"2023-11-01T04:16:01.755533+0000","last_unstale":"2023-11-01T04:16:01.756078+0000","last_undegraded":"2023-11-01T04:16:01.756078+0000","last_fullsized":"2023-11-01T04:16:01.756078+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:11.875831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:02.154377+0000","last_change":"2023-11-01T04:16:02.153588+0000","last_active":"2023-11-01T04:16:02.154377+0000","last_peered":"2023-11-01T04:16:02.154377+0000","last_clean":"2023-11-01T04:16:02.154377+0000","last_became_active":"2023-11-01T04:16:02.053124+0000","last_became_peered":"2023-11-01T04:16:02.053124+0000","last_unstale":"2023-11-01T04:16:02.154377+0000","last_undegraded":"2023-11-01T04:16:02.154377+0000","last_fullsized":"2023-11-01T04:16:02.154377+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:10.812663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:01.759172+0000","last_change":"2023-11-01T04:16:01.757411+0000","last_active":"2023-11-01T04:16:01.759172+0000","last_peered":"2023-11-01T04:16:01.759172+0000","last_clean":"2023-11-01T04:16:01.759172+0000","last_became_active":"2023-11-01T04:16:01.756620+0000","last_became_peered":"2023-11-01T04:16:01.756620+0000","last_unstale":"2023-11-01T04:16:01.759172+0000","last_undegraded":"2023-11-01T04:16:01.759172+0000","last_fullsized":"2023-11-01T04:16:01.759172+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:15.520106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:01.931313+0000","last_change":"2023-11-01T04:16:01.930714+0000","last_active":"2023-11-01T04:16:01.931313+0000","last_peered":"2023-11-01T04:16:01.931313+0000","last_clean":"2023-11-01T04:16:01.931313+0000","last_became_active":"2023-11-01T04:16:01.930333+0000","last_became_peered":"2023-11-01T04:16:01.930333+0000","last_unstale":"2023-11-01T04:16:01.931313+0000","last_undegraded":"2023-11-01T04:16:01.931313+0000","last_fullsized":"2023-11-01T04:16:01.931313+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:11.801695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:02.079266+0000","last_change":"2023-11-01T04:16:02.077904+0000","last_active":"2023-11-01T04:16:02.079266+0000","last_peered":"2023-11-01T04:16:02.079266+0000","last_clean":"2023-11-01T04:16:02.079266+0000","last_became_active":"2023-11-01T04:16:02.077101+0000","last_became_peered":"2023-11-01T04:16:02.077101+0000","last_unstale":"2023-11-01T04:16:02.079266+0000","last_undegraded":"2023-11-01T04:16:02.079266+0000","last_fullsized":"2023-11-01T04:16:02.079266+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:03.374641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"27'50","reported_seq":81,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:03.071402+0000","last_change":"2023-11-01T04:15:59.016143+0000","last_active":"2023-11-01T04:16:03.071402+0000","last_peered":"2023-11-01T04:16:03.071402+0000","last_clean":"2023-11-01T04:16:03.071402+0000","last_became_active":"2023-11-01T04:15:59.014137+0000","last_became_peered":"2023-11-01T04:15:59.014137+0000","last_unstale":"2023-11-01T04:16:03.071402+0000","last_undegraded":"2023-11-01T04:16:03.071402+0000","last_fullsized":"2023-11-01T04:16:03.071402+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_clean_scrub_stamp":"2023-11-01T04:15:57.686019+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:15.188172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":672800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":618496,"data_stored":445679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,6,7,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5029999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5940000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]}]},{"osd":14,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]}]},{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":618496,"data_stored":445679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6160000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.716}]}]},{"osd":6,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[5,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1000000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4119999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4490000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]}]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":118741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.153}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.093}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3849999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5539999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.625}]}]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.375}]}]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.345}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3899999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]}]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,5,6,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7250000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3600000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]}]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.121}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2050000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]}]},{"osd":7,"up_from":22,"seq":94489280515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,6,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.24}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4219999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]}]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27320,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":294912,"data_stored":118741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.464}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.869}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.696}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0819999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5950000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.988}]}]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":118741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5629999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.617}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.998}]}]},{"osd":11,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.274}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.454}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6419999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]}]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3320000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6439999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3839999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4219999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":336400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":336400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-01T04:16:04.733 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-01T04:16:05.015 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:16:05.016 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-11-01T04:16:05.031 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-11-01T04:16:03.604251+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":437612,"kb_used_data":5064,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509511828,"statfs":{"total":1546188226560,"available":1545740111872,"internally_reserved":0,"allocated":5185536,"data_stored":2449650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25505,"internal_metadata":442473567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":131072,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.837382"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:01.955150+0000","last_change":"2023-11-01T04:16:01.855207+0000","last_active":"2023-11-01T04:16:01.955150+0000","last_peered":"2023-11-01T04:16:01.955150+0000","last_clean":"2023-11-01T04:16:01.955150+0000","last_became_active":"2023-11-01T04:16:01.854811+0000","last_became_peered":"2023-11-01T04:16:01.854811+0000","last_unstale":"2023-11-01T04:16:01.955150+0000","last_undegraded":"2023-11-01T04:16:01.955150+0000","last_fullsized":"2023-11-01T04:16:01.955150+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:07.050216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-11-01T04:16:03.146049+0000","last_change":"2023-11-01T04:16:01.875964+0000","last_active":"2023-11-01T04:16:03.146049+0000","last_peered":"2023-11-01T04:16:03.146049+0000","last_clean":"2023-11-01T04:16:03.146049+0000","last_became_active":"2023-11-01T04:16:01.875255+0000","last_became_peered":"2023-11-01T04:16:01.875255+0000","last_unstale":"2023-11-01T04:16:03.146049+0000","last_undegraded":"2023-11-01T04:16:03.146049+0000","last_fullsized":"2023-11-01T04:16:03.146049+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:05.611210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11],"acting":[14,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:02.169746+0000","last_change":"2023-11-01T04:16:02.168355+0000","last_active":"2023-11-01T04:16:02.169746+0000","last_peered":"2023-11-01T04:16:02.169746+0000","last_clean":"2023-11-01T04:16:02.169746+0000","last_became_active":"2023-11-01T04:16:02.167473+0000","last_became_peered":"2023-11-01T04:16:02.167473+0000","last_unstale":"2023-11-01T04:16:02.169746+0000","last_undegraded":"2023-11-01T04:16:02.169746+0000","last_fullsized":"2023-11-01T04:16:02.169746+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:03.374641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:01.756078+0000","last_change":"2023-11-01T04:16:01.756078+0000","last_active":"2023-11-01T04:16:01.756078+0000","last_peered":"2023-11-01T04:16:01.756078+0000","last_clean":"2023-11-01T04:16:01.756078+0000","last_became_active":"2023-11-01T04:16:01.755533+0000","last_became_peered":"2023-11-01T04:16:01.755533+0000","last_unstale":"2023-11-01T04:16:01.756078+0000","last_undegraded":"2023-11-01T04:16:01.756078+0000","last_fullsized":"2023-11-01T04:16:01.756078+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:11.875831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:02.154377+0000","last_change":"2023-11-01T04:16:02.153588+0000","last_active":"2023-11-01T04:16:02.154377+0000","last_peered":"2023-11-01T04:16:02.154377+0000","last_clean":"2023-11-01T04:16:02.154377+0000","last_became_active":"2023-11-01T04:16:02.053124+0000","last_became_peered":"2023-11-01T04:16:02.053124+0000","last_unstale":"2023-11-01T04:16:02.154377+0000","last_undegraded":"2023-11-01T04:16:02.154377+0000","last_fullsized":"2023-11-01T04:16:02.154377+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:10.812663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:01.759172+0000","last_change":"2023-11-01T04:16:01.757411+0000","last_active":"2023-11-01T04:16:01.759172+0000","last_peered":"2023-11-01T04:16:01.759172+0000","last_clean":"2023-11-01T04:16:01.759172+0000","last_became_active":"2023-11-01T04:16:01.756620+0000","last_became_peered":"2023-11-01T04:16:01.756620+0000","last_unstale":"2023-11-01T04:16:01.759172+0000","last_undegraded":"2023-11-01T04:16:01.759172+0000","last_fullsized":"2023-11-01T04:16:01.759172+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:15.520106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:01.931313+0000","last_change":"2023-11-01T04:16:01.930714+0000","last_active":"2023-11-01T04:16:01.931313+0000","last_peered":"2023-11-01T04:16:01.931313+0000","last_clean":"2023-11-01T04:16:01.931313+0000","last_became_active":"2023-11-01T04:16:01.930333+0000","last_became_peered":"2023-11-01T04:16:01.930333+0000","last_unstale":"2023-11-01T04:16:01.931313+0000","last_undegraded":"2023-11-01T04:16:01.931313+0000","last_fullsized":"2023-11-01T04:16:01.931313+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:11.801695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:02.079266+0000","last_change":"2023-11-01T04:16:02.077904+0000","last_active":"2023-11-01T04:16:02.079266+0000","last_peered":"2023-11-01T04:16:02.079266+0000","last_clean":"2023-11-01T04:16:02.079266+0000","last_became_active":"2023-11-01T04:16:02.077101+0000","last_became_peered":"2023-11-01T04:16:02.077101+0000","last_unstale":"2023-11-01T04:16:02.079266+0000","last_undegraded":"2023-11-01T04:16:02.079266+0000","last_fullsized":"2023-11-01T04:16:02.079266+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:03.374641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"27'50","reported_seq":81,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:03.071402+0000","last_change":"2023-11-01T04:15:59.016143+0000","last_active":"2023-11-01T04:16:03.071402+0000","last_peered":"2023-11-01T04:16:03.071402+0000","last_clean":"2023-11-01T04:16:03.071402+0000","last_became_active":"2023-11-01T04:15:59.014137+0000","last_became_peered":"2023-11-01T04:15:59.014137+0000","last_unstale":"2023-11-01T04:16:03.071402+0000","last_undegraded":"2023-11-01T04:16:03.071402+0000","last_fullsized":"2023-11-01T04:16:03.071402+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_clean_scrub_stamp":"2023-11-01T04:15:57.686019+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:15.188172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":672800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":618496,"data_stored":445679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,6,7,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5029999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5940000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]}]},{"osd":14,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]}]},{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":618496,"data_stored":445679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6160000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.716}]}]},{"osd":6,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[5,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1000000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4119999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4490000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]}]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":118741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.153}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.093}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3849999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5539999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.625}]}]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.375}]}]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.345}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3899999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]}]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,5,6,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7250000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3600000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]}]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.121}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2050000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]}]},{"osd":7,"up_from":22,"seq":94489280515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,6,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.24}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4219999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]}]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27320,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":294912,"data_stored":118741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.464}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.869}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.696}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0819999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5950000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.988}]}]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":118741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5629999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.617}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.998}]}]},{"osd":11,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.274}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.454}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6419999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]}]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3320000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6439999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3839999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4219999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":336400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":336400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-01T04:16:05.033 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-01T04:16:05.033 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-01T04:16:05.034 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-01T04:16:05.034 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-01T04:16:05.325 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:16:05.325 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-01T04:16:05.341 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-01T04:16:05.342 INFO:teuthology.run_tasks:Running task thrashosds... 2023-11-01T04:16:05.352 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2023-11-01T04:16:05.352 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'DBG_maintain_prefix_itr_2'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'DBG_maintain_prefix_itr_2', 'sha1': '420cd129397301b1521ceb74e1bdbd9b855d2189'}} 2023-11-01T04:16:05.353 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2023-11-01T04:16:05.353 INFO:tasks.thrashosds:Beginning thrashosds... 2023-11-01T04:16:05.353 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-01T04:16:05.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-01T04:16:05.631 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:16:05.631 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-11-01T04:16:05.648 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-11-01T04:16:05.604936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":437612,"kb_used_data":5064,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509511828,"statfs":{"total":1546188226560,"available":1545740111872,"internally_reserved":0,"allocated":5185536,"data_stored":2449650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25505,"internal_metadata":442473567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":131072,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.838067"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:01.955150+0000","last_change":"2023-11-01T04:16:01.855207+0000","last_active":"2023-11-01T04:16:01.955150+0000","last_peered":"2023-11-01T04:16:01.955150+0000","last_clean":"2023-11-01T04:16:01.955150+0000","last_became_active":"2023-11-01T04:16:01.854811+0000","last_became_peered":"2023-11-01T04:16:01.854811+0000","last_unstale":"2023-11-01T04:16:01.955150+0000","last_undegraded":"2023-11-01T04:16:01.955150+0000","last_fullsized":"2023-11-01T04:16:01.955150+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:07.050216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-11-01T04:16:03.146049+0000","last_change":"2023-11-01T04:16:01.875964+0000","last_active":"2023-11-01T04:16:03.146049+0000","last_peered":"2023-11-01T04:16:03.146049+0000","last_clean":"2023-11-01T04:16:03.146049+0000","last_became_active":"2023-11-01T04:16:01.875255+0000","last_became_peered":"2023-11-01T04:16:01.875255+0000","last_unstale":"2023-11-01T04:16:03.146049+0000","last_undegraded":"2023-11-01T04:16:03.146049+0000","last_fullsized":"2023-11-01T04:16:03.146049+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:05.611210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11],"acting":[14,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:02.169746+0000","last_change":"2023-11-01T04:16:02.168355+0000","last_active":"2023-11-01T04:16:02.169746+0000","last_peered":"2023-11-01T04:16:02.169746+0000","last_clean":"2023-11-01T04:16:02.169746+0000","last_became_active":"2023-11-01T04:16:02.167473+0000","last_became_peered":"2023-11-01T04:16:02.167473+0000","last_unstale":"2023-11-01T04:16:02.169746+0000","last_undegraded":"2023-11-01T04:16:02.169746+0000","last_fullsized":"2023-11-01T04:16:02.169746+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:03.374641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:01.756078+0000","last_change":"2023-11-01T04:16:01.756078+0000","last_active":"2023-11-01T04:16:01.756078+0000","last_peered":"2023-11-01T04:16:01.756078+0000","last_clean":"2023-11-01T04:16:01.756078+0000","last_became_active":"2023-11-01T04:16:01.755533+0000","last_became_peered":"2023-11-01T04:16:01.755533+0000","last_unstale":"2023-11-01T04:16:01.756078+0000","last_undegraded":"2023-11-01T04:16:01.756078+0000","last_fullsized":"2023-11-01T04:16:01.756078+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:11.875831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:02.154377+0000","last_change":"2023-11-01T04:16:02.153588+0000","last_active":"2023-11-01T04:16:02.154377+0000","last_peered":"2023-11-01T04:16:02.154377+0000","last_clean":"2023-11-01T04:16:02.154377+0000","last_became_active":"2023-11-01T04:16:02.053124+0000","last_became_peered":"2023-11-01T04:16:02.053124+0000","last_unstale":"2023-11-01T04:16:02.154377+0000","last_undegraded":"2023-11-01T04:16:02.154377+0000","last_fullsized":"2023-11-01T04:16:02.154377+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:10.812663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:01.759172+0000","last_change":"2023-11-01T04:16:01.757411+0000","last_active":"2023-11-01T04:16:01.759172+0000","last_peered":"2023-11-01T04:16:01.759172+0000","last_clean":"2023-11-01T04:16:01.759172+0000","last_became_active":"2023-11-01T04:16:01.756620+0000","last_became_peered":"2023-11-01T04:16:01.756620+0000","last_unstale":"2023-11-01T04:16:01.759172+0000","last_undegraded":"2023-11-01T04:16:01.759172+0000","last_fullsized":"2023-11-01T04:16:01.759172+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:15.520106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:01.931313+0000","last_change":"2023-11-01T04:16:01.930714+0000","last_active":"2023-11-01T04:16:01.931313+0000","last_peered":"2023-11-01T04:16:01.931313+0000","last_clean":"2023-11-01T04:16:01.931313+0000","last_became_active":"2023-11-01T04:16:01.930333+0000","last_became_peered":"2023-11-01T04:16:01.930333+0000","last_unstale":"2023-11-01T04:16:01.931313+0000","last_undegraded":"2023-11-01T04:16:01.931313+0000","last_fullsized":"2023-11-01T04:16:01.931313+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:11.801695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:02.079266+0000","last_change":"2023-11-01T04:16:02.077904+0000","last_active":"2023-11-01T04:16:02.079266+0000","last_peered":"2023-11-01T04:16:02.079266+0000","last_clean":"2023-11-01T04:16:02.079266+0000","last_became_active":"2023-11-01T04:16:02.077101+0000","last_became_peered":"2023-11-01T04:16:02.077101+0000","last_unstale":"2023-11-01T04:16:02.079266+0000","last_undegraded":"2023-11-01T04:16:02.079266+0000","last_fullsized":"2023-11-01T04:16:02.079266+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:03.374641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"27'50","reported_seq":81,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:03.071402+0000","last_change":"2023-11-01T04:15:59.016143+0000","last_active":"2023-11-01T04:16:03.071402+0000","last_peered":"2023-11-01T04:16:03.071402+0000","last_clean":"2023-11-01T04:16:03.071402+0000","last_became_active":"2023-11-01T04:15:59.014137+0000","last_became_peered":"2023-11-01T04:15:59.014137+0000","last_unstale":"2023-11-01T04:16:03.071402+0000","last_undegraded":"2023-11-01T04:16:03.071402+0000","last_fullsized":"2023-11-01T04:16:03.071402+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_clean_scrub_stamp":"2023-11-01T04:15:57.686019+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:15.188172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":672800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":618496,"data_stored":445679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,6,7,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5029999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5940000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]}]},{"osd":14,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]}]},{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":618496,"data_stored":445679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6160000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.716}]}]},{"osd":6,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[5,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1000000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4119999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4490000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]}]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":118741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.153}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.093}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3849999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5539999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.625}]}]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.375}]}]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.345}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3899999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]}]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,5,6,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7250000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3600000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]}]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.121}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2050000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]}]},{"osd":7,"up_from":22,"seq":94489280515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,6,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.24}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4219999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]}]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27320,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":294912,"data_stored":118741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.464}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.869}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.696}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0819999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5950000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.988}]}]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":118741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5629999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.617}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.998}]}]},{"osd":11,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.274}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.454}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6419999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]}]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3320000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6439999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3839999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4219999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":336400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":336400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-01T04:16:05.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-01T04:16:05.935 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:16:05.936 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-11-01T04:16:05.952 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-11-01T04:16:05.604936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":437612,"kb_used_data":5064,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509511828,"statfs":{"total":1546188226560,"available":1545740111872,"internally_reserved":0,"allocated":5185536,"data_stored":2449650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25505,"internal_metadata":442473567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":131072,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.838067"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:01.955150+0000","last_change":"2023-11-01T04:16:01.855207+0000","last_active":"2023-11-01T04:16:01.955150+0000","last_peered":"2023-11-01T04:16:01.955150+0000","last_clean":"2023-11-01T04:16:01.955150+0000","last_became_active":"2023-11-01T04:16:01.854811+0000","last_became_peered":"2023-11-01T04:16:01.854811+0000","last_unstale":"2023-11-01T04:16:01.955150+0000","last_undegraded":"2023-11-01T04:16:01.955150+0000","last_fullsized":"2023-11-01T04:16:01.955150+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:07.050216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-11-01T04:16:03.146049+0000","last_change":"2023-11-01T04:16:01.875964+0000","last_active":"2023-11-01T04:16:03.146049+0000","last_peered":"2023-11-01T04:16:03.146049+0000","last_clean":"2023-11-01T04:16:03.146049+0000","last_became_active":"2023-11-01T04:16:01.875255+0000","last_became_peered":"2023-11-01T04:16:01.875255+0000","last_unstale":"2023-11-01T04:16:03.146049+0000","last_undegraded":"2023-11-01T04:16:03.146049+0000","last_fullsized":"2023-11-01T04:16:03.146049+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:05.611210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11],"acting":[14,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:02.169746+0000","last_change":"2023-11-01T04:16:02.168355+0000","last_active":"2023-11-01T04:16:02.169746+0000","last_peered":"2023-11-01T04:16:02.169746+0000","last_clean":"2023-11-01T04:16:02.169746+0000","last_became_active":"2023-11-01T04:16:02.167473+0000","last_became_peered":"2023-11-01T04:16:02.167473+0000","last_unstale":"2023-11-01T04:16:02.169746+0000","last_undegraded":"2023-11-01T04:16:02.169746+0000","last_fullsized":"2023-11-01T04:16:02.169746+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:03.374641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:01.756078+0000","last_change":"2023-11-01T04:16:01.756078+0000","last_active":"2023-11-01T04:16:01.756078+0000","last_peered":"2023-11-01T04:16:01.756078+0000","last_clean":"2023-11-01T04:16:01.756078+0000","last_became_active":"2023-11-01T04:16:01.755533+0000","last_became_peered":"2023-11-01T04:16:01.755533+0000","last_unstale":"2023-11-01T04:16:01.756078+0000","last_undegraded":"2023-11-01T04:16:01.756078+0000","last_fullsized":"2023-11-01T04:16:01.756078+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:11.875831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:02.154377+0000","last_change":"2023-11-01T04:16:02.153588+0000","last_active":"2023-11-01T04:16:02.154377+0000","last_peered":"2023-11-01T04:16:02.154377+0000","last_clean":"2023-11-01T04:16:02.154377+0000","last_became_active":"2023-11-01T04:16:02.053124+0000","last_became_peered":"2023-11-01T04:16:02.053124+0000","last_unstale":"2023-11-01T04:16:02.154377+0000","last_undegraded":"2023-11-01T04:16:02.154377+0000","last_fullsized":"2023-11-01T04:16:02.154377+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:10.812663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:01.759172+0000","last_change":"2023-11-01T04:16:01.757411+0000","last_active":"2023-11-01T04:16:01.759172+0000","last_peered":"2023-11-01T04:16:01.759172+0000","last_clean":"2023-11-01T04:16:01.759172+0000","last_became_active":"2023-11-01T04:16:01.756620+0000","last_became_peered":"2023-11-01T04:16:01.756620+0000","last_unstale":"2023-11-01T04:16:01.759172+0000","last_undegraded":"2023-11-01T04:16:01.759172+0000","last_fullsized":"2023-11-01T04:16:01.759172+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:15.520106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:01.931313+0000","last_change":"2023-11-01T04:16:01.930714+0000","last_active":"2023-11-01T04:16:01.931313+0000","last_peered":"2023-11-01T04:16:01.931313+0000","last_clean":"2023-11-01T04:16:01.931313+0000","last_became_active":"2023-11-01T04:16:01.930333+0000","last_became_peered":"2023-11-01T04:16:01.930333+0000","last_unstale":"2023-11-01T04:16:01.931313+0000","last_undegraded":"2023-11-01T04:16:01.931313+0000","last_fullsized":"2023-11-01T04:16:01.931313+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:11.801695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:02.079266+0000","last_change":"2023-11-01T04:16:02.077904+0000","last_active":"2023-11-01T04:16:02.079266+0000","last_peered":"2023-11-01T04:16:02.079266+0000","last_clean":"2023-11-01T04:16:02.079266+0000","last_became_active":"2023-11-01T04:16:02.077101+0000","last_became_peered":"2023-11-01T04:16:02.077101+0000","last_unstale":"2023-11-01T04:16:02.079266+0000","last_undegraded":"2023-11-01T04:16:02.079266+0000","last_fullsized":"2023-11-01T04:16:02.079266+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:03.374641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"27'50","reported_seq":81,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-11-01T04:16:03.071402+0000","last_change":"2023-11-01T04:15:59.016143+0000","last_active":"2023-11-01T04:16:03.071402+0000","last_peered":"2023-11-01T04:16:03.071402+0000","last_clean":"2023-11-01T04:16:03.071402+0000","last_became_active":"2023-11-01T04:15:59.014137+0000","last_became_peered":"2023-11-01T04:15:59.014137+0000","last_unstale":"2023-11-01T04:16:03.071402+0000","last_undegraded":"2023-11-01T04:16:03.071402+0000","last_fullsized":"2023-11-01T04:16:03.071402+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_clean_scrub_stamp":"2023-11-01T04:15:57.686019+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:15.188172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":81,"num_write_kb":848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":672800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50,"ondisk_log_size":50,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":618496,"data_stored":445679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,6,7,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.151}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.206}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3759999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5029999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5940000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]}]},{"osd":14,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,7,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2729999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5580000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.528}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]}]},{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":618496,"data_stored":445679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4590000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6160000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.716}]}]},{"osd":6,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[5,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1000000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4119999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4490000000000001}]}]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,9,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.234}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]}]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":118741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7470000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4179999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.153}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.093}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3849999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5539999999999998}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.625}]}]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.375}]}]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.345}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4419999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3899999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]}]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,5,6,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1659999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7250000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3600000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5189999999999999}]}]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.121}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2050000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5640000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]}]},{"osd":7,"up_from":22,"seq":94489280515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,6,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.24}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4219999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.27}]}]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27320,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":294912,"data_stored":118741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.464}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5209999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.869}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.903}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.696}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0819999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5950000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.988}]}]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":118741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5629999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.617}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.998}]}]},{"osd":11,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3220000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.274}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.454}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6419999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]}]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":278528,"data_stored":109279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,6,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2769999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3320000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6439999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3839999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4219999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":336400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":336400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-01T04:16:05.954 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-01T04:16:05.954 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2023-11-01T04:16:06.240 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 28 2023-11-01T04:16:06.240 INFO:teuthology.orchestra.run.smithi153.stdout:fsid 0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d 2023-11-01T04:16:06.240 INFO:teuthology.orchestra.run.smithi153.stdout:created 2023-11-01T04:15:44.902210+0000 2023-11-01T04:16:06.240 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2023-11-01T04:16:02.751700+0000 2023-11-01T04:16:06.241 INFO:teuthology.orchestra.run.smithi153.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-11-01T04:16:06.241 INFO:teuthology.orchestra.run.smithi153.stdout:crush_version 5 2023-11-01T04:16:06.241 INFO:teuthology.orchestra.run.smithi153.stdout:full_ratio 0.95 2023-11-01T04:16:06.241 INFO:teuthology.orchestra.run.smithi153.stdout:backfillfull_ratio 0.9 2023-11-01T04:16:06.241 INFO:teuthology.orchestra.run.smithi153.stdout:nearfull_ratio 0.85 2023-11-01T04:16:06.242 INFO:teuthology.orchestra.run.smithi153.stdout:require_min_compat_client luminous 2023-11-01T04:16:06.242 INFO:teuthology.orchestra.run.smithi153.stdout:min_compat_client jewel 2023-11-01T04:16:06.242 INFO:teuthology.orchestra.run.smithi153.stdout:require_osd_release reef 2023-11-01T04:16:06.242 INFO:teuthology.orchestra.run.smithi153.stdout:stretch_mode_enabled false 2023-11-01T04:16:06.242 INFO:teuthology.orchestra.run.smithi153.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.38 2023-11-01T04:16:06.242 INFO:teuthology.orchestra.run.smithi153.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 28 flags hashpspool stripe_width 0 application rbd read_balance_score 5.99 2023-11-01T04:16:06.243 INFO:teuthology.orchestra.run.smithi153.stdout:max_osd 16 2023-11-01T04:16:06.243 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6816/59702962,v1:172.21.15.153:6817/59702962] [v2:172.21.15.153:6818/59702962,v1:172.21.15.153:6819/59702962] exists,up 83ce3480-defe-4c73-aa1d-47c7941a23ba 2023-11-01T04:16:06.243 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6813/3103315277,v1:172.21.15.173:6815/3103315277] [v2:172.21.15.173:6817/3103315277,v1:172.21.15.173:6818/3103315277] exists,up 01a9801f-72fb-4299-8708-d70eb59b0fb2 2023-11-01T04:16:06.243 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6816/1072240131,v1:172.21.15.181:6817/1072240131] [v2:172.21.15.181:6818/1072240131,v1:172.21.15.181:6820/1072240131] exists,up 7c16db19-9bec-4ca0-88eb-bb5fb673d160 2023-11-01T04:16:06.243 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6826/2304315422,v1:172.21.15.190:6827/2304315422] [v2:172.21.15.190:6828/2304315422,v1:172.21.15.190:6829/2304315422] exists,up 4c4cc8d1-6761-4d20-90e8-7c3d43d889d8 2023-11-01T04:16:06.244 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6824/3209586931,v1:172.21.15.153:6825/3209586931] [v2:172.21.15.153:6826/3209586931,v1:172.21.15.153:6827/3209586931] exists,up 0fb14630-87e6-4456-9640-7986a5079821 2023-11-01T04:16:06.244 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6824/409879231,v1:172.21.15.173:6825/409879231] [v2:172.21.15.173:6826/409879231,v1:172.21.15.173:6827/409879231] exists,up cc715599-0461-477b-beb2-8777d6f36b22 2023-11-01T04:16:06.244 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6 up in weight 1 up_from 21 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6819/3465541770,v1:172.21.15.181:6822/3465541770] [v2:172.21.15.181:6826/3465541770,v1:172.21.15.181:6827/3465541770] exists,up ca5bea1c-f309-4258-b85d-738097037962 2023-11-01T04:16:06.244 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6802/1763196448,v1:172.21.15.190:6803/1763196448] [v2:172.21.15.190:6804/1763196448,v1:172.21.15.190:6805/1763196448] exists,up 1e80e965-2a46-468c-bcb8-49bee8dd19c1 2023-11-01T04:16:06.244 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6800/1068520389,v1:172.21.15.153:6801/1068520389] [v2:172.21.15.153:6802/1068520389,v1:172.21.15.153:6803/1068520389] exists,up 87179b74-818f-47df-ad70-1392cfe227e9 2023-11-01T04:16:06.244 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6800/210565898,v1:172.21.15.173:6801/210565898] [v2:172.21.15.173:6802/210565898,v1:172.21.15.173:6803/210565898] exists,up 10d3c443-1748-4a7c-8b60-b58cfd565b4d 2023-11-01T04:16:06.245 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6800/2007605188,v1:172.21.15.181:6801/2007605188] [v2:172.21.15.181:6802/2007605188,v1:172.21.15.181:6803/2007605188] exists,up 81bb6569-d02b-4638-8b76-f99756d415b5 2023-11-01T04:16:06.245 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6810/2456507663,v1:172.21.15.190:6812/2456507663] [v2:172.21.15.190:6814/2456507663,v1:172.21.15.190:6816/2456507663] exists,up b8aefe11-845f-48a4-930c-b05cdfbb460a 2023-11-01T04:16:06.245 INFO:teuthology.orchestra.run.smithi153.stdout:osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6808/819245857,v1:172.21.15.153:6809/819245857] [v2:172.21.15.153:6810/819245857,v1:172.21.15.153:6811/819245857] exists,up 2165437d-26ba-4e50-8646-8cedbc10da53 2023-11-01T04:16:06.245 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6808/3482699195,v1:172.21.15.173:6809/3482699195] [v2:172.21.15.173:6810/3482699195,v1:172.21.15.173:6811/3482699195] exists,up 7b9f38b4-f5ff-43f1-a373-2aaed710026d 2023-11-01T04:16:06.245 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6808/2780345165,v1:172.21.15.181:6809/2780345165] [v2:172.21.15.181:6810/2780345165,v1:172.21.15.181:6811/2780345165] exists,up 054b27d2-8a4d-4851-8af8-78b71e6a537e 2023-11-01T04:16:06.245 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6811/3871145029,v1:172.21.15.190:6813/3871145029] [v2:172.21.15.190:6815/3871145029,v1:172.21.15.190:6817/3871145029] exists,up e650f936-3712-47cd-9b17-d732991a27a3 2023-11-01T04:16:06.258 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6816/59702962,v1:172.21.15.153:6817/59702962] [v2:172.21.15.153:6818/59702962,v1:172.21.15.153:6819/59702962] exists,up 83ce3480-defe-4c73-aa1d-47c7941a23ba', 'osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6813/3103315277,v1:172.21.15.173:6815/3103315277] [v2:172.21.15.173:6817/3103315277,v1:172.21.15.173:6818/3103315277] exists,up 01a9801f-72fb-4299-8708-d70eb59b0fb2', 'osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6816/1072240131,v1:172.21.15.181:6817/1072240131] [v2:172.21.15.181:6818/1072240131,v1:172.21.15.181:6820/1072240131] exists,up 7c16db19-9bec-4ca0-88eb-bb5fb673d160', 'osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6826/2304315422,v1:172.21.15.190:6827/2304315422] [v2:172.21.15.190:6828/2304315422,v1:172.21.15.190:6829/2304315422] exists,up 4c4cc8d1-6761-4d20-90e8-7c3d43d889d8', 'osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6824/3209586931,v1:172.21.15.153:6825/3209586931] [v2:172.21.15.153:6826/3209586931,v1:172.21.15.153:6827/3209586931] exists,up 0fb14630-87e6-4456-9640-7986a5079821', 'osd.5 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6824/409879231,v1:172.21.15.173:6825/409879231] [v2:172.21.15.173:6826/409879231,v1:172.21.15.173:6827/409879231] exists,up cc715599-0461-477b-beb2-8777d6f36b22', 'osd.6 up in weight 1 up_from 21 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6819/3465541770,v1:172.21.15.181:6822/3465541770] [v2:172.21.15.181:6826/3465541770,v1:172.21.15.181:6827/3465541770] exists,up ca5bea1c-f309-4258-b85d-738097037962', 'osd.7 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6802/1763196448,v1:172.21.15.190:6803/1763196448] [v2:172.21.15.190:6804/1763196448,v1:172.21.15.190:6805/1763196448] exists,up 1e80e965-2a46-468c-bcb8-49bee8dd19c1', 'osd.8 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6800/1068520389,v1:172.21.15.153:6801/1068520389] [v2:172.21.15.153:6802/1068520389,v1:172.21.15.153:6803/1068520389] exists,up 87179b74-818f-47df-ad70-1392cfe227e9', 'osd.9 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6800/210565898,v1:172.21.15.173:6801/210565898] [v2:172.21.15.173:6802/210565898,v1:172.21.15.173:6803/210565898] exists,up 10d3c443-1748-4a7c-8b60-b58cfd565b4d', 'osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6800/2007605188,v1:172.21.15.181:6801/2007605188] [v2:172.21.15.181:6802/2007605188,v1:172.21.15.181:6803/2007605188] exists,up 81bb6569-d02b-4638-8b76-f99756d415b5', 'osd.11 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6810/2456507663,v1:172.21.15.190:6812/2456507663] [v2:172.21.15.190:6814/2456507663,v1:172.21.15.190:6816/2456507663] exists,up b8aefe11-845f-48a4-930c-b05cdfbb460a', 'osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6808/819245857,v1:172.21.15.153:6809/819245857] [v2:172.21.15.153:6810/819245857,v1:172.21.15.153:6811/819245857] exists,up 2165437d-26ba-4e50-8646-8cedbc10da53', 'osd.13 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6808/3482699195,v1:172.21.15.173:6809/3482699195] [v2:172.21.15.173:6810/3482699195,v1:172.21.15.173:6811/3482699195] exists,up 7b9f38b4-f5ff-43f1-a373-2aaed710026d', 'osd.14 up in weight 1 up_from 22 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6808/2780345165,v1:172.21.15.181:6809/2780345165] [v2:172.21.15.181:6810/2780345165,v1:172.21.15.181:6811/2780345165] exists,up 054b27d2-8a4d-4851-8af8-78b71e6a537e', 'osd.15 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6811/3871145029,v1:172.21.15.190:6813/3871145029] [v2:172.21.15.190:6815/3871145029,v1:172.21.15.190:6817/3871145029] exists,up e650f936-3712-47cd-9b17-d732991a27a3'] 2023-11-01T04:16:06.259 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2023-11-01T04:16:06.370 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:06.370 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "mon.a", 2023-11-01T04:16:06.370 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster": "ceph", 2023-11-01T04:16:06.370 INFO:teuthology.orchestra.run.smithi153.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2023-11-01T04:16:06.371 INFO:teuthology.orchestra.run.smithi153.stdout: "admin_socket_mode": "", 2023-11-01T04:16:06.371 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2023-11-01T04:16:06.371 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_client_required": "cephx, none", 2023-11-01T04:16:06.371 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_cluster_required": "cephx", 2023-11-01T04:16:06.371 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_debug": "true", 2023-11-01T04:16:06.372 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2023-11-01T04:16:06.372 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_mon_ticket_ttl": "660.000000", 2023-11-01T04:16:06.372 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_service_required": "cephx", 2023-11-01T04:16:06.372 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_service_ticket_ttl": "240.000000", 2023-11-01T04:16:06.372 INFO:teuthology.orchestra.run.smithi153.stdout: "auth_supported": "cephx", 2023-11-01T04:16:06.372 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_aio": "true", 2023-11-01T04:16:06.373 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_aio_max_queue_depth": "1024", 2023-11-01T04:16:06.373 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_aio_poll_ms": "250", 2023-11-01T04:16:06.373 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_aio_reap_max": "16", 2023-11-01T04:16:06.373 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_async_discard": "false", 2023-11-01T04:16:06.373 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_block_size": "4096", 2023-11-01T04:16:06.374 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_debug_aio": "false", 2023-11-01T04:16:06.374 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_debug_aio_log_age": "5.000000", 2023-11-01T04:16:06.374 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2023-11-01T04:16:06.374 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_debug_inflight_ios": "false", 2023-11-01T04:16:06.374 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_enable_discard": "false", 2023-11-01T04:16:06.375 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_flock_retry": "3", 2023-11-01T04:16:06.375 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_flock_retry_interval": "0.100000", 2023-11-01T04:16:06.375 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_inject_crash": "0", 2023-11-01T04:16:06.375 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_inject_crash_flush_delay": "2", 2023-11-01T04:16:06.375 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_ioring": "false", 2023-11-01T04:16:06.375 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_ioring_hipri": "false", 2023-11-01T04:16:06.376 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_ioring_sqthread_poll": "false", 2023-11-01T04:16:06.376 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_nvme_unbind_from_kernel": "false", 2023-11-01T04:16:06.376 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_read_buffer_alignment": "4096", 2023-11-01T04:16:06.376 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_read_preallocated_huge_buffers": "", 2023-11-01T04:16:06.376 INFO:teuthology.orchestra.run.smithi153.stdout: "bdev_type": "", 2023-11-01T04:16:06.376 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_alloc_size": "1048576", 2023-11-01T04:16:06.377 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_allocator": "hybrid", 2023-11-01T04:16:06.377 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_buffered_io": "true", 2023-11-01T04:16:06.377 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_check_for_zeros": "false", 2023-11-01T04:16:06.377 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_check_volume_selector_often": "false", 2023-11-01T04:16:06.377 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_check_volume_selector_on_umount": "false", 2023-11-01T04:16:06.378 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_compact_log_sync": "false", 2023-11-01T04:16:06.378 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2023-11-01T04:16:06.378 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2023-11-01T04:16:06.378 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_log_compact_min_size": "16777216", 2023-11-01T04:16:06.378 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_log_replay_check_allocations": "true", 2023-11-01T04:16:06.378 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_max_log_runway": "4194304", 2023-11-01T04:16:06.379 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_max_prefetch": "1048576", 2023-11-01T04:16:06.379 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_min_flush_size": "524288", 2023-11-01T04:16:06.379 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_min_log_runway": "1048576", 2023-11-01T04:16:06.379 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_replay_recovery": "false", 2023-11-01T04:16:06.379 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_replay_recovery_disable_compact": "false", 2023-11-01T04:16:06.379 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_shared_alloc_size": "65536", 2023-11-01T04:16:06.380 INFO:teuthology.orchestra.run.smithi153.stdout: "bluefs_sync_write": "false", 2023-11-01T04:16:06.380 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2023-11-01T04:16:06.380 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2023-11-01T04:16:06.380 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2023-11-01T04:16:06.380 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_allocation_from_file": "true", 2023-11-01T04:16:06.380 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_allocator": "hybrid", 2023-11-01T04:16:06.381 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2023-11-01T04:16:06.381 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2023-11-01T04:16:06.381 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2023-11-01T04:16:06.381 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2023-11-01T04:16:06.381 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2023-11-01T04:16:06.382 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_bitmapallocator_span_size": "1024", 2023-11-01T04:16:06.382 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_blobid_prealloc": "10240", 2023-11-01T04:16:06.382 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_create": "true", 2023-11-01T04:16:06.382 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_db_create": "false", 2023-11-01T04:16:06.382 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_db_path": "", 2023-11-01T04:16:06.383 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_db_size": "0", 2023-11-01T04:16:06.383 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_path": "", 2023-11-01T04:16:06.383 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_preallocate_file": "false", 2023-11-01T04:16:06.383 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_size": "107374182400", 2023-11-01T04:16:06.383 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_wal_create": "false", 2023-11-01T04:16:06.383 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_wal_path": "", 2023-11-01T04:16:06.384 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_block_wal_size": "100663296", 2023-11-01T04:16:06.384 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_bluefs": "true", 2023-11-01T04:16:06.384 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2023-11-01T04:16:06.384 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_bluefs_env_mirror": "false", 2023-11-01T04:16:06.384 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_bluefs_max_free": "10737418240", 2023-11-01T04:16:06.384 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2023-11-01T04:16:06.385 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2023-11-01T04:16:06.385 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2023-11-01T04:16:06.385 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2023-11-01T04:16:06.385 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2023-11-01T04:16:06.385 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_autotune": "true", 2023-11-01T04:16:06.385 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_autotune_interval": "5.000000", 2023-11-01T04:16:06.386 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2023-11-01T04:16:06.386 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_kv_ratio": "0.450000", 2023-11-01T04:16:06.386 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_meta_ratio": "0.450000", 2023-11-01T04:16:06.386 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_size": "0", 2023-11-01T04:16:06.386 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_size_hdd": "1073741824", 2023-11-01T04:16:06.387 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_size_ssd": "3221225472", 2023-11-01T04:16:06.387 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_trim_interval": "0.050000", 2023-11-01T04:16:06.387 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2023-11-01T04:16:06.387 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cache_type": "2q", 2023-11-01T04:16:06.387 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2023-11-01T04:16:06.387 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_clone_cow": "true", 2023-11-01T04:16:06.388 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_algorithm": "snappy", 2023-11-01T04:16:06.388 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_max_blob_size": "0", 2023-11-01T04:16:06.388 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2023-11-01T04:16:06.388 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2023-11-01T04:16:06.388 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_min_blob_size": "0", 2023-11-01T04:16:06.388 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2023-11-01T04:16:06.389 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2023-11-01T04:16:06.389 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_mode": "none", 2023-11-01T04:16:06.389 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_compression_required_ratio": "0.875000", 2023-11-01T04:16:06.389 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_csum_type": "crc32c", 2023-11-01T04:16:06.389 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_enforce_settings": "default", 2023-11-01T04:16:06.390 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_freelist": "false", 2023-11-01T04:16:06.390 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_fsck_abort": "false", 2023-11-01T04:16:06.390 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2023-11-01T04:16:06.390 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2023-11-01T04:16:06.390 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_inject_read_err": "false", 2023-11-01T04:16:06.391 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_legacy_omap": "false", 2023-11-01T04:16:06.391 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_no_reuse_blocks": "false", 2023-11-01T04:16:06.391 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_omit_block_device_write": "false", 2023-11-01T04:16:06.391 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_omit_kv_commit": "false", 2023-11-01T04:16:06.391 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2023-11-01T04:16:06.392 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_prefill": "0.000000", 2023-11-01T04:16:06.392 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_prefragment_max": "1048576", 2023-11-01T04:16:06.392 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_random_read_err": "0.000000", 2023-11-01T04:16:06.392 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2023-11-01T04:16:06.392 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_small_allocations": "0", 2023-11-01T04:16:06.392 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2023-11-01T04:16:06.393 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_default_buffered_read": "true", 2023-11-01T04:16:06.393 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_default_buffered_write": "false", 2023-11-01T04:16:06.393 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_deferred_batch_ops": "0", 2023-11-01T04:16:06.393 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2023-11-01T04:16:06.393 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2023-11-01T04:16:06.394 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_elastic_shared_blobs": "true", 2023-11-01T04:16:06.394 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2023-11-01T04:16:06.394 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_map_shard_max_size": "1200", 2023-11-01T04:16:06.394 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_map_shard_min_size": "150", 2023-11-01T04:16:06.394 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_map_shard_target_size": "500", 2023-11-01T04:16:06.394 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2023-11-01T04:16:06.395 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fail_eio": "false", 2023-11-01T04:16:06.395 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_freelist_blocks_per_key": "128", 2023-11-01T04:16:06.395 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2023-11-01T04:16:06.395 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2023-11-01T04:16:06.395 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2023-11-01T04:16:06.395 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_on_mkfs": "true", 2023-11-01T04:16:06.396 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2023-11-01T04:16:06.396 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_on_mount": "false", 2023-11-01T04:16:06.396 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_on_mount_deep": "false", 2023-11-01T04:16:06.396 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_on_umount": "false", 2023-11-01T04:16:06.396 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_on_umount_deep": "false", 2023-11-01T04:16:06.396 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2023-11-01T04:16:06.397 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_quick_fix_threads": "2", 2023-11-01T04:16:06.397 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2023-11-01T04:16:06.397 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2023-11-01T04:16:06.397 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_gc_enable_blob_threshold": "0", 2023-11-01T04:16:06.397 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_gc_enable_total_threshold": "0", 2023-11-01T04:16:06.398 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2023-11-01T04:16:06.398 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_ignore_data_csum": "false", 2023-11-01T04:16:06.398 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2023-11-01T04:16:06.398 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_kvbackend": "rocksdb", 2023-11-01T04:16:06.398 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_log_collection_list_age": "60.000000", 2023-11-01T04:16:06.398 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2023-11-01T04:16:06.399 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_log_op_age": "5.000000", 2023-11-01T04:16:06.399 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_max_alloc_size": "0", 2023-11-01T04:16:06.399 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_max_blob_size": "0", 2023-11-01T04:16:06.399 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_max_blob_size_hdd": "65536", 2023-11-01T04:16:06.399 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_max_blob_size_ssd": "65536", 2023-11-01T04:16:06.399 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_max_defer_interval": "3.000000", 2023-11-01T04:16:06.400 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_max_deferred_txc": "32", 2023-11-01T04:16:06.400 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_min_alloc_size": "0", 2023-11-01T04:16:06.400 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_min_alloc_size_hdd": "4096", 2023-11-01T04:16:06.400 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_min_alloc_size_ssd": "4096", 2023-11-01T04:16:06.400 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_nid_prealloc": "1024", 2023-11-01T04:16:06.400 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_prefer_deferred_size": "0", 2023-11-01T04:16:06.401 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2023-11-01T04:16:06.401 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2023-11-01T04:16:06.401 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_qfsck_on_mount": "true", 2023-11-01T04:16:06.401 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_retry_disk_reads": "3", 2023-11-01T04:16:06.401 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_rocksdb_cf": "true", 2023-11-01T04:16:06.401 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2023-11-01T04:16:06.402 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2023-11-01T04:16:06.402 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_rocksdb_options_annex": "", 2023-11-01T04:16:06.402 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_spdk_coremask": "0x1", 2023-11-01T04:16:06.402 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_spdk_io_sleep": "5", 2023-11-01T04:16:06.403 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_spdk_max_io_completion": "0", 2023-11-01T04:16:06.403 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_spdk_mem": "512", 2023-11-01T04:16:06.403 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_sync_submit_transaction": "false", 2023-11-01T04:16:06.403 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_throttle_bytes": "67108864", 2023-11-01T04:16:06.403 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_throttle_cost_per_io": "0", 2023-11-01T04:16:06.404 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2023-11-01T04:16:06.404 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2023-11-01T04:16:06.404 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2023-11-01T04:16:06.404 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_throttle_trace_rate": "0.000000", 2023-11-01T04:16:06.404 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_tracing": "false", 2023-11-01T04:16:06.404 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2023-11-01T04:16:06.405 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2023-11-01T04:16:06.405 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_volume_selection_reserved": "0", 2023-11-01T04:16:06.405 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2023-11-01T04:16:06.405 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2023-11-01T04:16:06.405 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_warn_on_legacy_statfs": "true", 2023-11-01T04:16:06.406 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2023-11-01T04:16:06.406 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2023-11-01T04:16:06.406 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2023-11-01T04:16:06.406 INFO:teuthology.orchestra.run.smithi153.stdout: "bluestore_zero_block_detection": "false", 2023-11-01T04:16:06.406 INFO:teuthology.orchestra.run.smithi153.stdout: "cephadm_path": "/usr/sbin/cephadm", 2023-11-01T04:16:06.407 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_action_update_interval": "2", 2023-11-01T04:16:06.407 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_directory_scan_interval": "10", 2023-11-01T04:16:06.407 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2023-11-01T04:16:06.407 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2023-11-01T04:16:06.408 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2023-11-01T04:16:06.408 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_mount_timeout": "10", 2023-11-01T04:16:06.408 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2023-11-01T04:16:06.408 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2023-11-01T04:16:06.408 INFO:teuthology.orchestra.run.smithi153.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2023-11-01T04:16:06.408 INFO:teuthology.orchestra.run.smithi153.stdout: "cephsqlite_blocklist_dead_locker": "true", 2023-11-01T04:16:06.409 INFO:teuthology.orchestra.run.smithi153.stdout: "cephsqlite_lock_renewal_interval": "2000", 2023-11-01T04:16:06.409 INFO:teuthology.orchestra.run.smithi153.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2023-11-01T04:16:06.409 INFO:teuthology.orchestra.run.smithi153.stdout: "cephx_cluster_require_signatures": "false", 2023-11-01T04:16:06.409 INFO:teuthology.orchestra.run.smithi153.stdout: "cephx_cluster_require_version": "2", 2023-11-01T04:16:06.409 INFO:teuthology.orchestra.run.smithi153.stdout: "cephx_require_signatures": "false", 2023-11-01T04:16:06.410 INFO:teuthology.orchestra.run.smithi153.stdout: "cephx_require_version": "2", 2023-11-01T04:16:06.410 INFO:teuthology.orchestra.run.smithi153.stdout: "cephx_service_require_signatures": "false", 2023-11-01T04:16:06.410 INFO:teuthology.orchestra.run.smithi153.stdout: "cephx_service_require_version": "2", 2023-11-01T04:16:06.410 INFO:teuthology.orchestra.run.smithi153.stdout: "cephx_sign_messages": "true", 2023-11-01T04:16:06.410 INFO:teuthology.orchestra.run.smithi153.stdout: "chdir": "", 2023-11-01T04:16:06.411 INFO:teuthology.orchestra.run.smithi153.stdout: "client_acl_type": "", 2023-11-01T04:16:06.411 INFO:teuthology.orchestra.run.smithi153.stdout: "client_asio_thread_count": "2", 2023-11-01T04:16:06.411 INFO:teuthology.orchestra.run.smithi153.stdout: "client_cache_mid": "0.750000", 2023-11-01T04:16:06.411 INFO:teuthology.orchestra.run.smithi153.stdout: "client_cache_size": "16384", 2023-11-01T04:16:06.411 INFO:teuthology.orchestra.run.smithi153.stdout: "client_caps_release_delay": "5", 2023-11-01T04:16:06.411 INFO:teuthology.orchestra.run.smithi153.stdout: "client_check_pool_perm": "true", 2023-11-01T04:16:06.412 INFO:teuthology.orchestra.run.smithi153.stdout: "client_collect_and_send_global_metrics": "false", 2023-11-01T04:16:06.412 INFO:teuthology.orchestra.run.smithi153.stdout: "client_debug_force_sync_read": "false", 2023-11-01T04:16:06.412 INFO:teuthology.orchestra.run.smithi153.stdout: "client_debug_getattr_caps": "false", 2023-11-01T04:16:06.412 INFO:teuthology.orchestra.run.smithi153.stdout: "client_debug_inject_tick_delay": "0", 2023-11-01T04:16:06.412 INFO:teuthology.orchestra.run.smithi153.stdout: "client_die_on_failed_dentry_invalidate": "true", 2023-11-01T04:16:06.413 INFO:teuthology.orchestra.run.smithi153.stdout: "client_die_on_failed_remount": "false", 2023-11-01T04:16:06.413 INFO:teuthology.orchestra.run.smithi153.stdout: "client_dirsize_rbytes": "true", 2023-11-01T04:16:06.413 INFO:teuthology.orchestra.run.smithi153.stdout: "client_force_lazyio": "false", 2023-11-01T04:16:06.413 INFO:teuthology.orchestra.run.smithi153.stdout: "client_fs": "", 2023-11-01T04:16:06.413 INFO:teuthology.orchestra.run.smithi153.stdout: "client_inject_fixed_oldest_tid": "false", 2023-11-01T04:16:06.413 INFO:teuthology.orchestra.run.smithi153.stdout: "client_inject_release_failure": "false", 2023-11-01T04:16:06.414 INFO:teuthology.orchestra.run.smithi153.stdout: "client_max_inline_size": "4096", 2023-11-01T04:16:06.414 INFO:teuthology.orchestra.run.smithi153.stdout: "client_max_retries_on_remount_failure": "5", 2023-11-01T04:16:06.414 INFO:teuthology.orchestra.run.smithi153.stdout: "client_mds_namespace": "", 2023-11-01T04:16:06.414 INFO:teuthology.orchestra.run.smithi153.stdout: "client_metadata": "", 2023-11-01T04:16:06.414 INFO:teuthology.orchestra.run.smithi153.stdout: "client_mount_gid": "-1", 2023-11-01T04:16:06.414 INFO:teuthology.orchestra.run.smithi153.stdout: "client_mount_timeout": "300", 2023-11-01T04:16:06.415 INFO:teuthology.orchestra.run.smithi153.stdout: "client_mount_uid": "-1", 2023-11-01T04:16:06.415 INFO:teuthology.orchestra.run.smithi153.stdout: "client_mountpoint": "/", 2023-11-01T04:16:06.415 INFO:teuthology.orchestra.run.smithi153.stdout: "client_notify_timeout": "10", 2023-11-01T04:16:06.415 INFO:teuthology.orchestra.run.smithi153.stdout: "client_oc": "true", 2023-11-01T04:16:06.415 INFO:teuthology.orchestra.run.smithi153.stdout: "client_oc_max_dirty": "104857600", 2023-11-01T04:16:06.416 INFO:teuthology.orchestra.run.smithi153.stdout: "client_oc_max_dirty_age": "5.000000", 2023-11-01T04:16:06.416 INFO:teuthology.orchestra.run.smithi153.stdout: "client_oc_max_objects": "1000", 2023-11-01T04:16:06.416 INFO:teuthology.orchestra.run.smithi153.stdout: "client_oc_size": "209715200", 2023-11-01T04:16:06.416 INFO:teuthology.orchestra.run.smithi153.stdout: "client_oc_target_dirty": "8388608", 2023-11-01T04:16:06.416 INFO:teuthology.orchestra.run.smithi153.stdout: "client_permissions": "true", 2023-11-01T04:16:06.417 INFO:teuthology.orchestra.run.smithi153.stdout: "client_quota": "true", 2023-11-01T04:16:06.417 INFO:teuthology.orchestra.run.smithi153.stdout: "client_quota_df": "true", 2023-11-01T04:16:06.417 INFO:teuthology.orchestra.run.smithi153.stdout: "client_readahead_max_bytes": "0", 2023-11-01T04:16:06.417 INFO:teuthology.orchestra.run.smithi153.stdout: "client_readahead_max_periods": "4", 2023-11-01T04:16:06.418 INFO:teuthology.orchestra.run.smithi153.stdout: "client_readahead_min": "131072", 2023-11-01T04:16:06.418 INFO:teuthology.orchestra.run.smithi153.stdout: "client_reconnect_stale": "false", 2023-11-01T04:16:06.418 INFO:teuthology.orchestra.run.smithi153.stdout: "client_shutdown_timeout": "30", 2023-11-01T04:16:06.418 INFO:teuthology.orchestra.run.smithi153.stdout: "client_snapdir": ".snap", 2023-11-01T04:16:06.418 INFO:teuthology.orchestra.run.smithi153.stdout: "client_tick_interval": "1", 2023-11-01T04:16:06.418 INFO:teuthology.orchestra.run.smithi153.stdout: "client_trace": "", 2023-11-01T04:16:06.419 INFO:teuthology.orchestra.run.smithi153.stdout: "client_try_dentry_invalidate": "false", 2023-11-01T04:16:06.419 INFO:teuthology.orchestra.run.smithi153.stdout: "client_use_faked_inos": "false", 2023-11-01T04:16:06.419 INFO:teuthology.orchestra.run.smithi153.stdout: "client_use_random_mds": "false", 2023-11-01T04:16:06.419 INFO:teuthology.orchestra.run.smithi153.stdout: "clog_to_graylog": "false", 2023-11-01T04:16:06.419 INFO:teuthology.orchestra.run.smithi153.stdout: "clog_to_graylog_host": "127.0.0.1", 2023-11-01T04:16:06.419 INFO:teuthology.orchestra.run.smithi153.stdout: "clog_to_graylog_port": "12201", 2023-11-01T04:16:06.420 INFO:teuthology.orchestra.run.smithi153.stdout: "clog_to_monitors": "default=true", 2023-11-01T04:16:06.420 INFO:teuthology.orchestra.run.smithi153.stdout: "clog_to_syslog": "false", 2023-11-01T04:16:06.420 INFO:teuthology.orchestra.run.smithi153.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2023-11-01T04:16:06.420 INFO:teuthology.orchestra.run.smithi153.stdout: "clog_to_syslog_level": "info", 2023-11-01T04:16:06.420 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "-", 2023-11-01T04:16:06.421 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_network": "", 2023-11-01T04:16:06.421 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_network_interface": "", 2023-11-01T04:16:06.421 INFO:teuthology.orchestra.run.smithi153.stdout: "compressor_zlib_isal": "false", 2023-11-01T04:16:06.421 INFO:teuthology.orchestra.run.smithi153.stdout: "compressor_zlib_level": "5", 2023-11-01T04:16:06.421 INFO:teuthology.orchestra.run.smithi153.stdout: "compressor_zlib_winsize": "-15", 2023-11-01T04:16:06.421 INFO:teuthology.orchestra.run.smithi153.stdout: "compressor_zstd_level": "1", 2023-11-01T04:16:06.422 INFO:teuthology.orchestra.run.smithi153.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2023-11-01T04:16:06.422 INFO:teuthology.orchestra.run.smithi153.stdout: "crash_dir": "/var/lib/ceph/crash", 2023-11-01T04:16:06.422 INFO:teuthology.orchestra.run.smithi153.stdout: "crimson_alien_op_num_threads": "6", 2023-11-01T04:16:06.422 INFO:teuthology.orchestra.run.smithi153.stdout: "crimson_alien_thread_cpu_cores": "", 2023-11-01T04:16:06.422 INFO:teuthology.orchestra.run.smithi153.stdout: "crimson_osd_obc_lru_size": "10", 2023-11-01T04:16:06.422 INFO:teuthology.orchestra.run.smithi153.stdout: "crimson_osd_scheduler_concurrency": "0", 2023-11-01T04:16:06.423 INFO:teuthology.orchestra.run.smithi153.stdout: "crimson_seastar_smp": "1", 2023-11-01T04:16:06.423 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "", 2023-11-01T04:16:06.423 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location_hook": "", 2023-11-01T04:16:06.423 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location_hook_timeout": "10", 2023-11-01T04:16:06.423 INFO:teuthology.orchestra.run.smithi153.stdout: "daemonize": "false", 2023-11-01T04:16:06.424 INFO:teuthology.orchestra.run.smithi153.stdout: "daos_pool": "tank", 2023-11-01T04:16:06.424 INFO:teuthology.orchestra.run.smithi153.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2023-11-01T04:16:06.424 INFO:teuthology.orchestra.run.smithi153.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2023-11-01T04:16:06.424 INFO:teuthology.orchestra.run.smithi153.stdout: "dbstore_db_name_prefix": "dbstore", 2023-11-01T04:16:06.424 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_alienstore": "0/5", 2023-11-01T04:16:06.424 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_allow_any_pool_priority": "false", 2023-11-01T04:16:06.425 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_asok": "1/5", 2023-11-01T04:16:06.425 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_asok_assert_abort": "false", 2023-11-01T04:16:06.425 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_asserts_on_shutdown": "true", 2023-11-01T04:16:06.425 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_auth": "20/20", 2023-11-01T04:16:06.425 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_bdev": "1/3", 2023-11-01T04:16:06.426 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_bluefs": "1/5", 2023-11-01T04:16:06.426 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_bluestore": "1/5", 2023-11-01T04:16:06.426 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_buffer": "0/1", 2023-11-01T04:16:06.426 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_ceph_exporter": "1/5", 2023-11-01T04:16:06.426 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_cephfs_mirror": "0/5", 2023-11-01T04:16:06.427 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_cephsqlite": "0/5", 2023-11-01T04:16:06.427 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_client": "0/5", 2023-11-01T04:16:06.427 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_compressor": "1/5", 2023-11-01T04:16:06.427 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_context": "0/1", 2023-11-01T04:16:06.427 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_crush": "1/1", 2023-11-01T04:16:06.427 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_crypto": "1/5", 2023-11-01T04:16:06.428 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_cyanstore": "0/5", 2023-11-01T04:16:06.428 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_deliberately_leak_memory": "false", 2023-11-01T04:16:06.428 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_disable_randomized_ping": "false", 2023-11-01T04:16:06.428 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_dpdk": "1/5", 2023-11-01T04:16:06.428 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_eventtrace": "1/5", 2023-11-01T04:16:06.428 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_filer": "0/1", 2023-11-01T04:16:06.429 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_filestore": "1/3", 2023-11-01T04:16:06.429 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_finisher": "1/1", 2023-11-01T04:16:06.429 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_fuse": "1/5", 2023-11-01T04:16:06.429 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_heartbeat_testing_span": "0", 2023-11-01T04:16:06.429 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_heartbeatmap": "1/5", 2023-11-01T04:16:06.429 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_immutable_obj_cache": "0/5", 2023-11-01T04:16:06.430 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_javaclient": "1/5", 2023-11-01T04:16:06.430 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_journal": "1/3", 2023-11-01T04:16:06.430 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_journaler": "0/5", 2023-11-01T04:16:06.430 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_kstore": "1/5", 2023-11-01T04:16:06.430 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_lockdep": "0/1", 2023-11-01T04:16:06.431 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mclock": "1/5", 2023-11-01T04:16:06.431 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mds": "1/5", 2023-11-01T04:16:06.431 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mds_balancer": "1/5", 2023-11-01T04:16:06.431 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mds_locker": "1/5", 2023-11-01T04:16:06.431 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mds_log": "1/5", 2023-11-01T04:16:06.431 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mds_log_expire": "1/5", 2023-11-01T04:16:06.432 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mds_migrator": "1/5", 2023-11-01T04:16:06.432 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_memstore": "1/5", 2023-11-01T04:16:06.432 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mgr": "2/5", 2023-11-01T04:16:06.432 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mgrc": "1/5", 2023-11-01T04:16:06.432 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_mon": "20/20", 2023-11-01T04:16:06.433 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_monc": "0/10", 2023-11-01T04:16:06.433 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_ms": "1/1", 2023-11-01T04:16:06.433 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_none": "0/5", 2023-11-01T04:16:06.433 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_objclass": "0/5", 2023-11-01T04:16:06.433 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_objectcacher": "0/5", 2023-11-01T04:16:06.433 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_objecter": "0/1", 2023-11-01T04:16:06.434 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_optracker": "0/5", 2023-11-01T04:16:06.434 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_osd": "1/5", 2023-11-01T04:16:06.434 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_paxos": "20/20", 2023-11-01T04:16:06.434 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_perfcounter": "1/5", 2023-11-01T04:16:06.434 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_prioritycache": "1/5", 2023-11-01T04:16:06.434 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rados": "0/5", 2023-11-01T04:16:06.435 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rbd": "0/5", 2023-11-01T04:16:06.435 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rbd_mirror": "0/5", 2023-11-01T04:16:06.435 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rbd_pwl": "0/5", 2023-11-01T04:16:06.435 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rbd_replay": "0/5", 2023-11-01T04:16:06.435 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_refs": "0/0", 2023-11-01T04:16:06.435 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_reserver": "1/1", 2023-11-01T04:16:06.436 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rgw": "1/5", 2023-11-01T04:16:06.436 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rgw_access": "1/5", 2023-11-01T04:16:06.436 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rgw_datacache": "1/5", 2023-11-01T04:16:06.436 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rgw_dbstore": "1/5", 2023-11-01T04:16:06.436 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rgw_flight": "1/5", 2023-11-01T04:16:06.436 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rgw_sync": "1/5", 2023-11-01T04:16:06.437 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_rocksdb": "4/5", 2023-11-01T04:16:06.437 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore": "0/5", 2023-11-01T04:16:06.437 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_backref": "0/5", 2023-11-01T04:16:06.437 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_cache": "0/5", 2023-11-01T04:16:06.437 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_cleaner": "0/5", 2023-11-01T04:16:06.437 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_device": "0/5", 2023-11-01T04:16:06.438 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_epm": "0/5", 2023-11-01T04:16:06.438 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_fixedkv_tree": "0/5", 2023-11-01T04:16:06.438 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_journal": "0/5", 2023-11-01T04:16:06.438 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_lba": "0/5", 2023-11-01T04:16:06.438 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_odata": "0/5", 2023-11-01T04:16:06.439 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_omap": "0/5", 2023-11-01T04:16:06.439 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_onode": "0/5", 2023-11-01T04:16:06.439 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_t": "0/5", 2023-11-01T04:16:06.439 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_seastore_tm": "0/5", 2023-11-01T04:16:06.439 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_striper": "0/1", 2023-11-01T04:16:06.439 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_test": "0/5", 2023-11-01T04:16:06.440 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_throttle": "1/1", 2023-11-01T04:16:06.440 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_timer": "0/1", 2023-11-01T04:16:06.440 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_tp": "0/5", 2023-11-01T04:16:06.440 INFO:teuthology.orchestra.run.smithi153.stdout: "debug_trace": "1/5", 2023-11-01T04:16:06.440 INFO:teuthology.orchestra.run.smithi153.stdout: "defer_client_eviction_on_laggy_osds": "true", 2023-11-01T04:16:06.440 INFO:teuthology.orchestra.run.smithi153.stdout: "device_failure_prediction_mode": "none", 2023-11-01T04:16:06.441 INFO:teuthology.orchestra.run.smithi153.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2023-11-01T04:16:06.441 INFO:teuthology.orchestra.run.smithi153.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2023-11-01T04:16:06.441 INFO:teuthology.orchestra.run.smithi153.stdout: "err_to_graylog": "false", 2023-11-01T04:16:06.441 INFO:teuthology.orchestra.run.smithi153.stdout: "err_to_journald": "false", 2023-11-01T04:16:06.441 INFO:teuthology.orchestra.run.smithi153.stdout: "err_to_stderr": "true", 2023-11-01T04:16:06.442 INFO:teuthology.orchestra.run.smithi153.stdout: "err_to_syslog": "false", 2023-11-01T04:16:06.442 INFO:teuthology.orchestra.run.smithi153.stdout: "event_tracing": "false", 2023-11-01T04:16:06.442 INFO:teuthology.orchestra.run.smithi153.stdout: "exporter_addr": "0.0.0.0", 2023-11-01T04:16:06.442 INFO:teuthology.orchestra.run.smithi153.stdout: "exporter_http_port": "9926", 2023-11-01T04:16:06.442 INFO:teuthology.orchestra.run.smithi153.stdout: "exporter_prio_limit": "5", 2023-11-01T04:16:06.442 INFO:teuthology.orchestra.run.smithi153.stdout: "exporter_sock_dir": "/var/run/ceph/", 2023-11-01T04:16:06.443 INFO:teuthology.orchestra.run.smithi153.stdout: "exporter_sort_metrics": "true", 2023-11-01T04:16:06.443 INFO:teuthology.orchestra.run.smithi153.stdout: "exporter_stats_period": "5", 2023-11-01T04:16:06.443 INFO:teuthology.orchestra.run.smithi153.stdout: "fake_statfs_for_testing": "0", 2023-11-01T04:16:06.443 INFO:teuthology.orchestra.run.smithi153.stdout: "fatal_signal_handlers": "true", 2023-11-01T04:16:06.443 INFO:teuthology.orchestra.run.smithi153.stdout: "filer_max_purge_ops": "10", 2023-11-01T04:16:06.443 INFO:teuthology.orchestra.run.smithi153.stdout: "filer_max_truncate_ops": "128", 2023-11-01T04:16:06.444 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_apply_finisher_threads": "1", 2023-11-01T04:16:06.444 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_blackhole": "false", 2023-11-01T04:16:06.444 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_btrfs_clone_range": "true", 2023-11-01T04:16:06.444 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_btrfs_snap": "true", 2023-11-01T04:16:06.444 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_caller_concurrency": "10", 2023-11-01T04:16:06.445 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_collect_device_partition_information": "true", 2023-11-01T04:16:06.445 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_commit_timeout": "600.000000", 2023-11-01T04:16:06.445 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_debug_inject_read_err": "false", 2023-11-01T04:16:06.445 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_debug_omap_check": "false", 2023-11-01T04:16:06.445 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_debug_random_read_err": "0.000000", 2023-11-01T04:16:06.446 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_debug_verify_split": "false", 2023-11-01T04:16:06.446 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_dump_file": "", 2023-11-01T04:16:06.446 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2023-11-01T04:16:06.446 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_expected_throughput_ops": "200.000000", 2023-11-01T04:16:06.446 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_fadvise": "true", 2023-11-01T04:16:06.446 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_fail_eio": "true", 2023-11-01T04:16:06.447 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_fd_cache_shards": "16", 2023-11-01T04:16:06.447 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_fd_cache_size": "128", 2023-11-01T04:16:06.447 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_fiemap": "false", 2023-11-01T04:16:06.447 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_fiemap_threshold": "4096", 2023-11-01T04:16:06.447 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_fsync_flushes_journal_data": "false", 2023-11-01T04:16:06.447 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_index_retry_probability": "0.000000", 2023-11-01T04:16:06.448 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_inject_stall": "0", 2023-11-01T04:16:06.448 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_journal_parallel": "false", 2023-11-01T04:16:06.448 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_journal_trailing": "false", 2023-11-01T04:16:06.448 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_journal_writeahead": "false", 2023-11-01T04:16:06.448 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_kill_at": "0", 2023-11-01T04:16:06.448 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_alloc_hint_size": "1048576", 2023-11-01T04:16:06.449 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_inline_xattr_size": "0", 2023-11-01T04:16:06.449 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2023-11-01T04:16:06.449 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_inline_xattr_size_other": "512", 2023-11-01T04:16:06.449 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2023-11-01T04:16:06.449 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_inline_xattrs": "0", 2023-11-01T04:16:06.450 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2023-11-01T04:16:06.450 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_inline_xattrs_other": "2", 2023-11-01T04:16:06.450 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_inline_xattrs_xfs": "10", 2023-11-01T04:16:06.450 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_sync_interval": "5.000000", 2023-11-01T04:16:06.450 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_xattr_value_size": "0", 2023-11-01T04:16:06.450 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2023-11-01T04:16:06.451 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_xattr_value_size_other": "1024", 2023-11-01T04:16:06.451 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2023-11-01T04:16:06.451 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_merge_threshold": "-10", 2023-11-01T04:16:06.451 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_min_sync_interval": "0.010000", 2023-11-01T04:16:06.451 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_odsync_write": "false", 2023-11-01T04:16:06.451 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_omap_backend": "rocksdb", 2023-11-01T04:16:06.452 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_omap_backend_path": "", 2023-11-01T04:16:06.452 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_omap_header_cache_size": "1024", 2023-11-01T04:16:06.452 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_ondisk_finisher_threads": "1", 2023-11-01T04:16:06.452 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_op_thread_suicide_timeout": "180", 2023-11-01T04:16:06.452 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_op_thread_timeout": "60", 2023-11-01T04:16:06.453 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_op_threads": "2", 2023-11-01T04:16:06.453 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_punch_hole": "false", 2023-11-01T04:16:06.453 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2023-11-01T04:16:06.453 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2023-11-01T04:16:06.453 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2023-11-01T04:16:06.453 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_high_threshhold": "0.900000", 2023-11-01T04:16:06.454 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_low_threshhold": "0.300000", 2023-11-01T04:16:06.454 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_max_bytes": "104857600", 2023-11-01T04:16:06.454 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2023-11-01T04:16:06.454 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2023-11-01T04:16:06.454 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2023-11-01T04:16:06.454 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_queue_max_ops": "50", 2023-11-01T04:16:06.455 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2023-11-01T04:16:06.455 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_seek_data_hole": "false", 2023-11-01T04:16:06.455 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_sloppy_crc": "false", 2023-11-01T04:16:06.455 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_sloppy_crc_block_size": "65536", 2023-11-01T04:16:06.455 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_splice": "false", 2023-11-01T04:16:06.455 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_split_multiple": "2", 2023-11-01T04:16:06.456 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_split_rand_factor": "20", 2023-11-01T04:16:06.456 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_update_to": "1000", 2023-11-01T04:16:06.456 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2023-11-01T04:16:06.456 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2023-11-01T04:16:06.456 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2023-11-01T04:16:06.456 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2023-11-01T04:16:06.457 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2023-11-01T04:16:06.457 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2023-11-01T04:16:06.457 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_enable": "true", 2023-11-01T04:16:06.457 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2023-11-01T04:16:06.457 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2023-11-01T04:16:06.457 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2023-11-01T04:16:06.458 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2023-11-01T04:16:06.458 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2023-11-01T04:16:06.458 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2023-11-01T04:16:06.458 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_xfs_extsize": "false", 2023-11-01T04:16:06.458 INFO:teuthology.orchestra.run.smithi153.stdout: "filestore_zfs_snap": "false", 2023-11-01T04:16:06.459 INFO:teuthology.orchestra.run.smithi153.stdout: "fio_dir": "/tmp/fio", 2023-11-01T04:16:06.459 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d", 2023-11-01T04:16:06.459 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_allow_other": "true", 2023-11-01T04:16:06.459 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_atomic_o_trunc": "true", 2023-11-01T04:16:06.459 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_big_writes": "true", 2023-11-01T04:16:06.459 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_debug": "false", 2023-11-01T04:16:06.460 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_default_permissions": "false", 2023-11-01T04:16:06.460 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_disable_pagecache": "false", 2023-11-01T04:16:06.460 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_max_write": "0", 2023-11-01T04:16:06.460 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_multithreaded": "true", 2023-11-01T04:16:06.460 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_require_active_mds": "true", 2023-11-01T04:16:06.461 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_set_user_groups": "true", 2023-11-01T04:16:06.461 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_splice_move": "true", 2023-11-01T04:16:06.461 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_splice_read": "true", 2023-11-01T04:16:06.461 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_splice_write": "true", 2023-11-01T04:16:06.461 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_syncfs_on_mksnap": "true", 2023-11-01T04:16:06.462 INFO:teuthology.orchestra.run.smithi153.stdout: "fuse_use_invalidate_cb": "true", 2023-11-01T04:16:06.462 INFO:teuthology.orchestra.run.smithi153.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2023-11-01T04:16:06.462 INFO:teuthology.orchestra.run.smithi153.stdout: "gss_target_name": "ceph", 2023-11-01T04:16:06.462 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_file": "", 2023-11-01T04:16:06.462 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_inject_failure": "0", 2023-11-01T04:16:06.462 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_interval": "5", 2023-11-01T04:16:06.463 INFO:teuthology.orchestra.run.smithi153.stdout: "host": "smithi153", 2023-11-01T04:16:06.463 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2023-11-01T04:16:06.463 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_max_inflight_ops": "128", 2023-11-01T04:16:06.463 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_max_size": "1073741824", 2023-11-01T04:16:06.463 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2023-11-01T04:16:06.463 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_qos_bps_burst": "0", 2023-11-01T04:16:06.464 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2023-11-01T04:16:06.464 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_qos_bps_limit": "0", 2023-11-01T04:16:06.464 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_qos_iops_burst": "0", 2023-11-01T04:16:06.464 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2023-11-01T04:16:06.464 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_qos_iops_limit": "0", 2023-11-01T04:16:06.464 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2023-11-01T04:16:06.465 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2023-11-01T04:16:06.465 INFO:teuthology.orchestra.run.smithi153.stdout: "immutable_object_cache_watermark": "0.900000", 2023-11-01T04:16:06.465 INFO:teuthology.orchestra.run.smithi153.stdout: "inject_early_sigterm": "false", 2023-11-01T04:16:06.465 INFO:teuthology.orchestra.run.smithi153.stdout: "jaeger_agent_port": "6799", 2023-11-01T04:16:06.465 INFO:teuthology.orchestra.run.smithi153.stdout: "jaeger_tracing_enable": "false", 2023-11-01T04:16:06.465 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_aio": "true", 2023-11-01T04:16:06.466 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_align_min_size": "65536", 2023-11-01T04:16:06.466 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_block_align": "true", 2023-11-01T04:16:06.466 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_block_size": "4096", 2023-11-01T04:16:06.466 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_dio": "true", 2023-11-01T04:16:06.466 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_discard": "false", 2023-11-01T04:16:06.466 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_force_aio": "false", 2023-11-01T04:16:06.467 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_ignore_corruption": "false", 2023-11-01T04:16:06.467 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_max_write_bytes": "10485760", 2023-11-01T04:16:06.467 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_max_write_entries": "100", 2023-11-01T04:16:06.467 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_replay_from": "0", 2023-11-01T04:16:06.467 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_throttle_high_multiple": "0.000000", 2023-11-01T04:16:06.468 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_throttle_high_threshhold": "0.900000", 2023-11-01T04:16:06.468 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_throttle_low_threshhold": "0.600000", 2023-11-01T04:16:06.468 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_throttle_max_multiple": "0.000000", 2023-11-01T04:16:06.468 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_write_header_frequency": "0", 2023-11-01T04:16:06.468 INFO:teuthology.orchestra.run.smithi153.stdout: "journal_zero_on_create": "false", 2023-11-01T04:16:06.468 INFO:teuthology.orchestra.run.smithi153.stdout: "journaler_prefetch_periods": "10", 2023-11-01T04:16:06.469 INFO:teuthology.orchestra.run.smithi153.stdout: "journaler_prezero_periods": "5", 2023-11-01T04:16:06.469 INFO:teuthology.orchestra.run.smithi153.stdout: "journaler_write_head_interval": "15", 2023-11-01T04:16:06.469 INFO:teuthology.orchestra.run.smithi153.stdout: "key": "", 2023-11-01T04:16:06.469 INFO:teuthology.orchestra.run.smithi153.stdout: "keyfile": "", 2023-11-01T04:16:06.469 INFO:teuthology.orchestra.run.smithi153.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2023-11-01T04:16:06.469 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_backend": "rocksdb", 2023-11-01T04:16:06.470 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_default_stripe_size": "65536", 2023-11-01T04:16:06.470 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_fsck_on_mount": "false", 2023-11-01T04:16:06.470 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_fsck_on_mount_deep": "true", 2023-11-01T04:16:06.470 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_max_bytes": "67108864", 2023-11-01T04:16:06.470 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_max_ops": "512", 2023-11-01T04:16:06.470 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_nid_prealloc": "1024", 2023-11-01T04:16:06.471 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_onode_map_size": "1024", 2023-11-01T04:16:06.471 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2023-11-01T04:16:06.471 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_sync_submit_transaction": "false", 2023-11-01T04:16:06.471 INFO:teuthology.orchestra.run.smithi153.stdout: "kstore_sync_transaction": "false", 2023-11-01T04:16:06.473 INFO:teuthology.orchestra.run.smithi153.stdout: "librados_thread_count": "2", 2023-11-01T04:16:06.473 INFO:teuthology.orchestra.run.smithi153.stdout: "lockdep": "false", 2023-11-01T04:16:06.473 INFO:teuthology.orchestra.run.smithi153.stdout: "lockdep_force_backtrace": "false", 2023-11-01T04:16:06.473 INFO:teuthology.orchestra.run.smithi153.stdout: "log_coarse_timestamps": "true", 2023-11-01T04:16:06.473 INFO:teuthology.orchestra.run.smithi153.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2023-11-01T04:16:06.474 INFO:teuthology.orchestra.run.smithi153.stdout: "log_flush_on_exit": "false", 2023-11-01T04:16:06.474 INFO:teuthology.orchestra.run.smithi153.stdout: "log_graylog_host": "127.0.0.1", 2023-11-01T04:16:06.474 INFO:teuthology.orchestra.run.smithi153.stdout: "log_graylog_port": "12201", 2023-11-01T04:16:06.474 INFO:teuthology.orchestra.run.smithi153.stdout: "log_max_new": "1000", 2023-11-01T04:16:06.474 INFO:teuthology.orchestra.run.smithi153.stdout: "log_max_recent": "10000", 2023-11-01T04:16:06.474 INFO:teuthology.orchestra.run.smithi153.stdout: "log_stderr_prefix": "", 2023-11-01T04:16:06.475 INFO:teuthology.orchestra.run.smithi153.stdout: "log_stop_at_utilization": "0.970000", 2023-11-01T04:16:06.475 INFO:teuthology.orchestra.run.smithi153.stdout: "log_to_file": "true", 2023-11-01T04:16:06.475 INFO:teuthology.orchestra.run.smithi153.stdout: "log_to_graylog": "false", 2023-11-01T04:16:06.475 INFO:teuthology.orchestra.run.smithi153.stdout: "log_to_journald": "false", 2023-11-01T04:16:06.475 INFO:teuthology.orchestra.run.smithi153.stdout: "log_to_stderr": "false", 2023-11-01T04:16:06.475 INFO:teuthology.orchestra.run.smithi153.stdout: "log_to_syslog": "false", 2023-11-01T04:16:06.476 INFO:teuthology.orchestra.run.smithi153.stdout: "max_rotating_auth_attempts": "10", 2023-11-01T04:16:06.476 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2023-11-01T04:16:06.476 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_action_on_write_error": "1", 2023-11-01T04:16:06.476 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_alternate_name_max": "8192", 2023-11-01T04:16:06.476 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_asio_thread_count": "2", 2023-11-01T04:16:06.476 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_export_pin": "true", 2023-11-01T04:16:06.477 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_fragment_dirs": "true", 2023-11-01T04:16:06.477 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2023-11-01T04:16:06.477 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_fragment_interval": "5", 2023-11-01T04:16:06.477 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_fragment_size_max": "100000", 2023-11-01T04:16:06.477 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_idle_threshold": "0.000000", 2023-11-01T04:16:06.477 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_interval": "10", 2023-11-01T04:16:06.478 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_max": "-1", 2023-11-01T04:16:06.478 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_max_until": "-1", 2023-11-01T04:16:06.478 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_merge_size": "50", 2023-11-01T04:16:06.478 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_midchunk": "0.300000", 2023-11-01T04:16:06.478 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_min_rebalance": "0.100000", 2023-11-01T04:16:06.479 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_min_start": "0.200000", 2023-11-01T04:16:06.479 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_minchunk": "0.001000", 2023-11-01T04:16:06.479 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_mode": "0", 2023-11-01T04:16:06.479 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_need_max": "1.200000", 2023-11-01T04:16:06.479 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_need_min": "0.800000", 2023-11-01T04:16:06.480 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_overload_epochs": "2", 2023-11-01T04:16:06.480 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_replicate_threshold": "8000.000000", 2023-11-01T04:16:06.480 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_sample_interval": "3.000000", 2023-11-01T04:16:06.480 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_split_bits": "3", 2023-11-01T04:16:06.480 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_split_rd": "25000.000000", 2023-11-01T04:16:06.480 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_split_size": "10000", 2023-11-01T04:16:06.481 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_split_wr": "10000.000000", 2023-11-01T04:16:06.481 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_target_decay": "10.000000", 2023-11-01T04:16:06.481 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2023-11-01T04:16:06.481 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_beacon_grace": "15.000000", 2023-11-01T04:16:06.481 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_beacon_interval": "4.000000", 2023-11-01T04:16:06.482 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_beacon_mon_down_grace": "60", 2023-11-01T04:16:06.482 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_memory_limit": "4294967296", 2023-11-01T04:16:06.482 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_mid": "0.700000", 2023-11-01T04:16:06.482 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_release_free_interval": "10", 2023-11-01T04:16:06.482 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_reservation": "0.050000", 2023-11-01T04:16:06.482 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_trim_decay_rate": "1.000000", 2023-11-01T04:16:06.483 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_trim_interval": "1", 2023-11-01T04:16:06.483 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cache_trim_threshold": "262144", 2023-11-01T04:16:06.483 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2023-11-01T04:16:06.483 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2023-11-01T04:16:06.483 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_client_delegate_inos_pct": "50", 2023-11-01T04:16:06.483 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_client_prealloc_inos": "1000", 2023-11-01T04:16:06.484 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2023-11-01T04:16:06.484 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_connect_bootstrapping": "false", 2023-11-01T04:16:06.484 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_damage_table_max_entries": "10000", 2023-11-01T04:16:06.484 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2023-11-01T04:16:06.484 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_debug_auth_pins": "false", 2023-11-01T04:16:06.485 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_debug_frag": "false", 2023-11-01T04:16:06.485 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_debug_scatterstat": "false", 2023-11-01T04:16:06.485 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_debug_subtrees": "false", 2023-11-01T04:16:06.485 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_decay_halflife": "5.000000", 2023-11-01T04:16:06.485 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_default_dir_hash": "2", 2023-11-01T04:16:06.485 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_defer_session_stale": "true", 2023-11-01T04:16:06.486 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_deny_all_reconnect": "false", 2023-11-01T04:16:06.486 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dir_keys_per_op": "16384", 2023-11-01T04:16:06.486 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dir_max_commit_size": "10", 2023-11-01T04:16:06.486 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dir_max_entries": "0", 2023-11-01T04:16:06.486 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dir_prefetch": "true", 2023-11-01T04:16:06.486 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dirstat_min_interval": "1.000000", 2023-11-01T04:16:06.487 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dump_cache_after_rejoin": "false", 2023-11-01T04:16:06.487 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dump_cache_on_map": "false", 2023-11-01T04:16:06.487 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dump_cache_threshold_file": "0", 2023-11-01T04:16:06.487 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2023-11-01T04:16:06.487 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_early_reply": "true", 2023-11-01T04:16:06.488 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_enable_op_tracker": "true", 2023-11-01T04:16:06.488 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_enforce_unique_name": "true", 2023-11-01T04:16:06.488 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_export_ephemeral_distributed": "true", 2023-11-01T04:16:06.488 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2023-11-01T04:16:06.488 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_export_ephemeral_random": "true", 2023-11-01T04:16:06.488 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_export_ephemeral_random_max": "0.010000", 2023-11-01T04:16:06.489 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_extraordinary_events_dump_interval": "0", 2023-11-01T04:16:06.489 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_forward_all_requests_to_auth": "false", 2023-11-01T04:16:06.489 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_freeze_tree_timeout": "30.000000", 2023-11-01T04:16:06.489 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_fscrypt_last_block_max_size": "4096", 2023-11-01T04:16:06.489 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_go_bad_corrupt_dentry": "true", 2023-11-01T04:16:06.489 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2023-11-01T04:16:06.490 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_health_cache_threshold": "1.500000", 2023-11-01T04:16:06.490 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_health_summarize_threshold": "10", 2023-11-01T04:16:06.490 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_heartbeat_grace": "15.000000", 2023-11-01T04:16:06.490 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_heartbeat_reset_grace": "1000", 2023-11-01T04:16:06.491 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_inject_health_dummy": "false", 2023-11-01T04:16:06.491 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2023-11-01T04:16:06.491 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_inject_migrator_session_race": "false", 2023-11-01T04:16:06.491 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2023-11-01T04:16:06.491 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_inject_skip_replaying_inotable": "false", 2023-11-01T04:16:06.491 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2023-11-01T04:16:06.492 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_join_fs": "", 2023-11-01T04:16:06.492 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_journal_format": "1", 2023-11-01T04:16:06.492 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_create_at": "0", 2023-11-01T04:16:06.492 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_export_at": "0", 2023-11-01T04:16:06.492 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_import_at": "0", 2023-11-01T04:16:06.492 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_journal_at": "0", 2023-11-01T04:16:06.493 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_journal_expire_at": "0", 2023-11-01T04:16:06.493 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_journal_replay_at": "0", 2023-11-01T04:16:06.493 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_link_at": "0", 2023-11-01T04:16:06.493 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_mdstable_at": "0", 2023-11-01T04:16:06.493 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_openc_at": "0", 2023-11-01T04:16:06.494 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_rename_at": "0", 2023-11-01T04:16:06.494 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_shutdown_at": "0", 2023-11-01T04:16:06.494 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_kill_skip_replaying_inotable": "false", 2023-11-01T04:16:06.494 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_event_large_threshold": "524288", 2023-11-01T04:16:06.494 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_events_per_segment": "1024", 2023-11-01T04:16:06.494 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_major_segment_event_ratio": "12", 2023-11-01T04:16:06.495 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_max_events": "-1", 2023-11-01T04:16:06.495 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_max_segments": "128", 2023-11-01T04:16:06.495 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_pause": "false", 2023-11-01T04:16:06.495 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_segment_size": "0", 2023-11-01T04:16:06.495 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_skip_corrupt_events": "false", 2023-11-01T04:16:06.495 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_skip_unbounded_events": "false", 2023-11-01T04:16:06.496 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_log_warn_factor": "2.000000", 2023-11-01T04:16:06.496 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_caps_per_client": "1048576", 2023-11-01T04:16:06.496 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_completed_flushes": "100000", 2023-11-01T04:16:06.496 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_completed_requests": "100000", 2023-11-01T04:16:06.496 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_export_size": "20971520", 2023-11-01T04:16:06.496 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_file_recover": "32", 2023-11-01T04:16:06.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_purge_files": "64", 2023-11-01T04:16:06.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_purge_ops": "8192", 2023-11-01T04:16:06.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2023-11-01T04:16:06.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_scrub_ops_in_progress": "5", 2023-11-01T04:16:06.497 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_max_snaps_per_dir": "100", 2023-11-01T04:16:06.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_metrics_update_interval": "2", 2023-11-01T04:16:06.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_min_caps_per_client": "100", 2023-11-01T04:16:06.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_min_caps_working_set": "10000", 2023-11-01T04:16:06.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_mon_shutdown_timeout": "5.000000", 2023-11-01T04:16:06.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_numa_node": "-1", 2023-11-01T04:16:06.498 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_oft_prefetch_dirfrags": "false", 2023-11-01T04:16:06.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_op_complaint_time": "30.000000", 2023-11-01T04:16:06.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_op_history_duration": "600", 2023-11-01T04:16:06.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_op_history_size": "20", 2023-11-01T04:16:06.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_op_history_slow_op_size": "20", 2023-11-01T04:16:06.499 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_op_history_slow_op_threshold": "10", 2023-11-01T04:16:06.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_op_log_threshold": "5", 2023-11-01T04:16:06.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_ping_grace": "15", 2023-11-01T04:16:06.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_ping_interval": "5", 2023-11-01T04:16:06.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2023-11-01T04:16:06.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_recall_global_max_decay_threshold": "131072", 2023-11-01T04:16:06.500 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_recall_max_caps": "30000", 2023-11-01T04:16:06.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_recall_max_decay_rate": "1.500000", 2023-11-01T04:16:06.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_recall_max_decay_threshold": "131072", 2023-11-01T04:16:06.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_recall_warning_decay_rate": "60.000000", 2023-11-01T04:16:06.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_recall_warning_threshold": "262144", 2023-11-01T04:16:06.501 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_reconnect_timeout": "45.000000", 2023-11-01T04:16:06.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_replay_interval": "1.000000", 2023-11-01T04:16:06.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_replay_unsafe_with_closed_session": "false", 2023-11-01T04:16:06.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_request_load_average_decay_rate": "60.000000", 2023-11-01T04:16:06.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_root_ino_gid": "0", 2023-11-01T04:16:06.502 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_root_ino_uid": "0", 2023-11-01T04:16:06.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_scatter_nudge_interval": "5.000000", 2023-11-01T04:16:06.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_session_blocklist_on_evict": "true", 2023-11-01T04:16:06.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_session_blocklist_on_timeout": "true", 2023-11-01T04:16:06.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2023-11-01T04:16:06.503 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_session_cache_liveness_magnitude": "10", 2023-11-01T04:16:06.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2023-11-01T04:16:06.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_session_cap_acquisition_throttle": "100000", 2023-11-01T04:16:06.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2023-11-01T04:16:06.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_session_metadata_threshold": "16777216", 2023-11-01T04:16:06.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sessionmap_keys_per_op": "1024", 2023-11-01T04:16:06.504 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_shutdown_check": "0", 2023-11-01T04:16:06.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_skip_ino": "0", 2023-11-01T04:16:06.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_sleep_rank_change": "0.000000", 2023-11-01T04:16:06.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_snap_max_uid": "4294967294", 2023-11-01T04:16:06.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_snap_min_uid": "0", 2023-11-01T04:16:06.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_snap_rstat": "false", 2023-11-01T04:16:06.505 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_standby_replay_damaged": "false", 2023-11-01T04:16:06.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_symlink_recovery": "true", 2023-11-01T04:16:06.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_task_status_update_interval": "2.000000", 2023-11-01T04:16:06.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_thrash_exports": "0", 2023-11-01T04:16:06.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_thrash_fragments": "0", 2023-11-01T04:16:06.506 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_tick_interval": "5.000000", 2023-11-01T04:16:06.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_valgrind_exit": "false", 2023-11-01T04:16:06.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_verify_backtrace": "1", 2023-11-01T04:16:06.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_verify_scatter": "false", 2023-11-01T04:16:06.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_wipe_ino_prealloc": "false", 2023-11-01T04:16:06.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mds_wipe_sessions": "false", 2023-11-01T04:16:06.507 INFO:teuthology.orchestra.run.smithi153.stdout: "mempool_debug": "false", 2023-11-01T04:16:06.508 INFO:teuthology.orchestra.run.smithi153.stdout: "memstore_debug_omit_block_device_write": "false", 2023-11-01T04:16:06.508 INFO:teuthology.orchestra.run.smithi153.stdout: "memstore_device_bytes": "1073741824", 2023-11-01T04:16:06.508 INFO:teuthology.orchestra.run.smithi153.stdout: "memstore_page_set": "false", 2023-11-01T04:16:06.508 INFO:teuthology.orchestra.run.smithi153.stdout: "memstore_page_size": "65536", 2023-11-01T04:16:06.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_client_bytes": "134217728", 2023-11-01T04:16:06.508 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_client_messages": "512", 2023-11-01T04:16:06.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2023-11-01T04:16:06.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_connect_retry_interval": "1.000000", 2023-11-01T04:16:06.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2023-11-01T04:16:06.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2023-11-01T04:16:06.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_disabled_modules": "", 2023-11-01T04:16:06.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_initial_modules": "restful iostat nfs", 2023-11-01T04:16:06.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_max_pg_num_change": "128", 2023-11-01T04:16:06.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_mds_bytes": "134217728", 2023-11-01T04:16:06.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_mds_messages": "128", 2023-11-01T04:16:06.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2023-11-01T04:16:06.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_mon_bytes": "134217728", 2023-11-01T04:16:06.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_mon_messages": "128", 2023-11-01T04:16:06.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_osd_bytes": "536870912", 2023-11-01T04:16:06.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_osd_messages": "8192", 2023-11-01T04:16:06.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_pool": "true", 2023-11-01T04:16:06.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_service_beacon_grace": "60.000000", 2023-11-01T04:16:06.511 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_standby_modules": "true", 2023-11-01T04:16:06.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_stats_period": "5", 2023-11-01T04:16:06.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_stats_threshold": "5", 2023-11-01T04:16:06.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_tick_period": "2", 2023-11-01T04:16:06.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr_ttl_cache_expire_seconds": "0", 2023-11-01T04:16:06.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_accept_timeout_factor": "2.000000", 2023-11-01T04:16:06.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_allow_pool_delete": "true", 2023-11-01T04:16:06.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_allow_pool_size_one": "true", 2023-11-01T04:16:06.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_auth_validate_all_caps": "true", 2023-11-01T04:16:06.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2023-11-01T04:16:06.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2023-11-01T04:16:06.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_bytes": "104857600", 2023-11-01T04:16:06.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_directed_command_retry": "2", 2023-11-01T04:16:06.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_hunt_interval": "3.000000", 2023-11-01T04:16:06.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2023-11-01T04:16:06.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2023-11-01T04:16:06.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2023-11-01T04:16:06.514 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_hunt_parallel": "3", 2023-11-01T04:16:06.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_log_interval": "1.000000", 2023-11-01T04:16:06.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_max_log_entries_per_message": "1000", 2023-11-01T04:16:06.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_ping_interval": "10.000000", 2023-11-01T04:16:06.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_ping_timeout": "30.000000", 2023-11-01T04:16:06.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_client_target_rank": "-1", 2023-11-01T04:16:06.515 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_clock_drift_allowed": "1.000000", 2023-11-01T04:16:06.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2023-11-01T04:16:06.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2023-11-01T04:16:06.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_file_level": "debug", 2023-11-01T04:16:06.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_file": "true", 2023-11-01T04:16:06.516 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_graylog": "false", 2023-11-01T04:16:06.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2023-11-01T04:16:06.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_graylog_port": "12201", 2023-11-01T04:16:06.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_journald": "false", 2023-11-01T04:16:06.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_stderr": "false", 2023-11-01T04:16:06.517 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_syslog": "default=false", 2023-11-01T04:16:06.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2023-11-01T04:16:06.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cluster_log_to_syslog_level": "info", 2023-11-01T04:16:06.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_compact_on_bootstrap": "false", 2023-11-01T04:16:06.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_compact_on_start": "false", 2023-11-01T04:16:06.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_compact_on_trim": "true", 2023-11-01T04:16:06.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_con_tracker_persist_interval": "10", 2023-11-01T04:16:06.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_con_tracker_score_halflife": "43200", 2023-11-01T04:16:06.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_config_key_max_entry_size": "65536", 2023-11-01T04:16:06.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_cpu_threads": "4", 2023-11-01T04:16:06.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_crush_min_required_version": "hammer", 2023-11-01T04:16:06.519 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_daemon_bytes": "419430400", 2023-11-01T04:16:06.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2023-11-01T04:16:06.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_data_avail_crit": "5", 2023-11-01T04:16:06.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_data_avail_warn": "5", 2023-11-01T04:16:06.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_data_size_warn": "16106127360", 2023-11-01T04:16:06.520 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_block_osdmap_trim": "false", 2023-11-01T04:16:06.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_deprecated_as_obsolete": "false", 2023-11-01T04:16:06.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_dump_json": "false", 2023-11-01T04:16:06.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2023-11-01T04:16:06.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_dump_transactions": "false", 2023-11-01T04:16:06.521 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_extra_checks": "false", 2023-11-01T04:16:06.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_no_initial_persistent_features": "false", 2023-11-01T04:16:06.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2023-11-01T04:16:06.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_no_require_quincy": "false", 2023-11-01T04:16:06.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_no_require_reef": "false", 2023-11-01T04:16:06.522 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2023-11-01T04:16:06.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_delta_reset_interval": "10.000000", 2023-11-01T04:16:06.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_dns_srv_name": "ceph-mon", 2023-11-01T04:16:06.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_down_mkfs_grace": "120", 2023-11-01T04:16:06.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_election_default_strategy": "1", 2023-11-01T04:16:06.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_election_timeout": "5.000000", 2023-11-01T04:16:06.523 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2023-11-01T04:16:06.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_elector_ping_divisor": "2", 2023-11-01T04:16:06.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_elector_ping_timeout": "2.000000", 2023-11-01T04:16:06.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_enable_op_tracker": "true", 2023-11-01T04:16:06.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_fake_pool_delete": "false", 2023-11-01T04:16:06.524 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_force_quorum_join": "false", 2023-11-01T04:16:06.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_globalid_prealloc": "10000", 2023-11-01T04:16:06.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_health_detail_to_clog": "false", 2023-11-01T04:16:06.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_health_log_update_period": "5", 2023-11-01T04:16:06.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_health_max_detail": "50", 2023-11-01T04:16:06.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_health_to_clog": "true", 2023-11-01T04:16:06.525 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_health_to_clog_interval": "600", 2023-11-01T04:16:06.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2023-11-01T04:16:06.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_host": "172.21.15.153,172.21.15.173,172.21.15.181", 2023-11-01T04:16:06.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_host_override": "", 2023-11-01T04:16:06.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_initial_members": "", 2023-11-01T04:16:06.526 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2023-11-01T04:16:06.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2023-11-01T04:16:06.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_inject_transaction_delay_max": "10.000000", 2023-11-01T04:16:06.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2023-11-01T04:16:06.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_keyvaluedb": "rocksdb", 2023-11-01T04:16:06.527 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_lease": "5.000000", 2023-11-01T04:16:06.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2023-11-01T04:16:06.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_lease_renew_interval_factor": "0.600000", 2023-11-01T04:16:06.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_log_full_interval": "50", 2023-11-01T04:16:06.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_log_max": "10000", 2023-11-01T04:16:06.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_log_max_summary": "50", 2023-11-01T04:16:06.528 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_max_log_entries_per_event": "4096", 2023-11-01T04:16:06.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_max_log_epochs": "500", 2023-11-01T04:16:06.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_max_mdsmap_epochs": "500", 2023-11-01T04:16:06.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_max_mgrmap_epochs": "500", 2023-11-01T04:16:06.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_max_osd": "10000", 2023-11-01T04:16:06.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_max_pg_per_osd": "10000", 2023-11-01T04:16:06.529 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_max_pool_pg_num": "65536", 2023-11-01T04:16:06.530 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_max_snap_prune_per_epoch": "100", 2023-11-01T04:16:06.530 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mds_blocklist_interval": "86400.000000", 2023-11-01T04:16:06.530 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mds_force_trim_to": "0", 2023-11-01T04:16:06.530 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mds_skip_sanity": "false", 2023-11-01T04:16:06.530 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_memory_autotune": "true", 2023-11-01T04:16:06.531 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_memory_target": "2147483648", 2023-11-01T04:16:06.531 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mgr_beacon_grace": "30", 2023-11-01T04:16:06.531 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2023-11-01T04:16:06.531 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mgr_digest_period": "5", 2023-11-01T04:16:06.531 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mgr_inactive_grace": "60", 2023-11-01T04:16:06.531 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mgr_mkfs_grace": "240", 2023-11-01T04:16:06.532 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2023-11-01T04:16:06.532 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_min_osdmap_epochs": "500", 2023-11-01T04:16:06.532 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_op_complaint_time": "30", 2023-11-01T04:16:06.532 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_op_history_duration": "600", 2023-11-01T04:16:06.532 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_op_history_size": "20", 2023-11-01T04:16:06.532 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_op_history_slow_op_size": "20", 2023-11-01T04:16:06.533 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_op_history_slow_op_threshold": "10", 2023-11-01T04:16:06.533 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_op_log_threshold": "5", 2023-11-01T04:16:06.533 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_adjust_down_out_interval": "true", 2023-11-01T04:16:06.533 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2023-11-01T04:16:06.533 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2023-11-01T04:16:06.534 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_auto_mark_in": "false", 2023-11-01T04:16:06.534 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_auto_mark_new_in": "true", 2023-11-01T04:16:06.534 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2023-11-01T04:16:06.534 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2023-11-01T04:16:06.534 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_cache_size": "500", 2023-11-01T04:16:06.534 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_cache_size_min": "134217728", 2023-11-01T04:16:06.535 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_crush_smoke_test": "true", 2023-11-01T04:16:06.535 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_destroyed_out_interval": "600", 2023-11-01T04:16:06.535 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_down_out_interval": "600", 2023-11-01T04:16:06.535 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_down_out_subtree_limit": "rack", 2023-11-01T04:16:06.535 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2023-11-01T04:16:06.535 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_force_trim_to": "0", 2023-11-01T04:16:06.536 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_full_ratio": "0.950000", 2023-11-01T04:16:06.536 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2023-11-01T04:16:06.536 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_laggy_halflife": "3600", 2023-11-01T04:16:06.536 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_laggy_max_interval": "300", 2023-11-01T04:16:06.536 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_laggy_weight": "0.300000", 2023-11-01T04:16:06.536 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2023-11-01T04:16:06.537 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_max_creating_pgs": "1024", 2023-11-01T04:16:06.537 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_max_initial_pgs": "1024", 2023-11-01T04:16:06.537 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_min_down_reporters": "2", 2023-11-01T04:16:06.537 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_min_in_ratio": "0.750000", 2023-11-01T04:16:06.537 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_min_up_ratio": "0.300000", 2023-11-01T04:16:06.537 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_nearfull_ratio": "0.850000", 2023-11-01T04:16:06.538 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_prime_pg_temp": "true", 2023-11-01T04:16:06.538 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2023-11-01T04:16:06.538 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2023-11-01T04:16:06.538 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_report_timeout": "900", 2023-11-01T04:16:06.538 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_reporter_subtree_level": "osd", 2023-11-01T04:16:06.539 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2023-11-01T04:16:06.539 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_warn_num_repaired": "10", 2023-11-01T04:16:06.539 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osd_warn_op_age": "32.000000", 2023-11-01T04:16:06.539 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osdmap_full_prune_enabled": "true", 2023-11-01T04:16:06.539 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osdmap_full_prune_interval": "10", 2023-11-01T04:16:06.539 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osdmap_full_prune_min": "10000", 2023-11-01T04:16:06.540 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_osdmap_full_prune_txsize": "100", 2023-11-01T04:16:06.540 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2023-11-01T04:16:06.540 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_pg_stuck_threshold": "60", 2023-11-01T04:16:06.540 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2023-11-01T04:16:06.540 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_pg_warn_min_objects": "10000", 2023-11-01T04:16:06.540 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_pg_warn_min_per_osd": "0", 2023-11-01T04:16:06.541 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_pg_warn_min_pool_objects": "1000", 2023-11-01T04:16:06.541 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_pool_quota_crit_threshold": "0", 2023-11-01T04:16:06.541 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_pool_quota_warn_threshold": "0", 2023-11-01T04:16:06.541 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_probe_timeout": "2.000000", 2023-11-01T04:16:06.541 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_reweight_max_change": "0.050000", 2023-11-01T04:16:06.541 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_reweight_max_osds": "4", 2023-11-01T04:16:06.542 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_reweight_min_bytes_per_osd": "10", 2023-11-01T04:16:06.542 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_reweight_min_pgs_per_osd": "4", 2023-11-01T04:16:06.542 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2023-11-01T04:16:06.542 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2023-11-01T04:16:06.542 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2023-11-01T04:16:06.542 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_scrub_interval": "300", 2023-11-01T04:16:06.543 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_scrub_max_keys": "100", 2023-11-01T04:16:06.543 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_scrub_timeout": "300", 2023-11-01T04:16:06.543 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_session_timeout": "300", 2023-11-01T04:16:06.543 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_smart_report_timeout": "5", 2023-11-01T04:16:06.543 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_stat_smooth_intervals": "6", 2023-11-01T04:16:06.543 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2023-11-01T04:16:06.544 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_stretch_pool_min_size": "2", 2023-11-01T04:16:06.544 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_stretch_pool_size": "4", 2023-11-01T04:16:06.544 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2023-11-01T04:16:06.544 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_subscribe_interval": "86400.000000", 2023-11-01T04:16:06.544 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_sync_debug": "false", 2023-11-01T04:16:06.544 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_sync_max_payload_keys": "2000", 2023-11-01T04:16:06.545 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_sync_max_payload_size": "1048576", 2023-11-01T04:16:06.545 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_sync_provider_kill_at": "0", 2023-11-01T04:16:06.545 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_sync_requester_kill_at": "0", 2023-11-01T04:16:06.545 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_sync_timeout": "60.000000", 2023-11-01T04:16:06.545 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_target_pg_per_osd": "100", 2023-11-01T04:16:06.545 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_tick_interval": "5", 2023-11-01T04:16:06.546 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_timecheck_interval": "300.000000", 2023-11-01T04:16:06.546 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_timecheck_skew_interval": "30.000000", 2023-11-01T04:16:06.546 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_use_min_delay_socket": "false", 2023-11-01T04:16:06.546 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_older_version_delay": "604800", 2023-11-01T04:16:06.546 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2023-11-01T04:16:06.547 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2023-11-01T04:16:06.547 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2023-11-01T04:16:06.547 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_filestore_osds": "false", 2023-11-01T04:16:06.547 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2023-11-01T04:16:06.547 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2023-11-01T04:16:06.547 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2023-11-01T04:16:06.548 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_misplaced": "false", 2023-11-01T04:16:06.548 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2023-11-01T04:16:06.548 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_older_version": "true", 2023-11-01T04:16:06.548 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2023-11-01T04:16:06.548 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_pool_no_app": "true", 2023-11-01T04:16:06.548 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_pool_no_redundancy": "false", 2023-11-01T04:16:06.549 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2023-11-01T04:16:06.549 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2023-11-01T04:16:06.549 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2023-11-01T04:16:06.549 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_on_too_few_osds": "false", 2023-11-01T04:16:06.549 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2023-11-01T04:16:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2023-11-01T04:16:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": "", 2023-11-01T04:16:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2023-11-01T04:16:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2023-11-01T04:16:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2023-11-01T04:16:06.550 INFO:teuthology.orchestra.run.smithi153.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2023-11-01T04:16:06.551 INFO:teuthology.orchestra.run.smithi153.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2023-11-01T04:16:06.551 INFO:teuthology.orchestra.run.smithi153.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2023-11-01T04:16:06.551 INFO:teuthology.orchestra.run.smithi153.stdout: "motr_tracing_enabled": "false", 2023-11-01T04:16:06.551 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_op_threads": "3", 2023-11-01T04:16:06.551 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_buffer_size": "131072", 2023-11-01T04:16:06.551 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_cm": "false", 2023-11-01T04:16:06.552 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_device_name": "", 2023-11-01T04:16:06.552 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_dscp": "96", 2023-11-01T04:16:06.552 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_enable_hugepage": "false", 2023-11-01T04:16:06.552 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_gid_idx": "0", 2023-11-01T04:16:06.552 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_local_gid": "", 2023-11-01T04:16:06.553 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_polling_us": "1000", 2023-11-01T04:16:06.553 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_port_num": "1", 2023-11-01T04:16:06.553 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_receive_buffers": "32768", 2023-11-01T04:16:06.553 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_receive_queue_len": "4096", 2023-11-01T04:16:06.553 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_roce_ver": "1", 2023-11-01T04:16:06.553 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_send_buffers": "1024", 2023-11-01T04:16:06.554 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_sl": "3", 2023-11-01T04:16:06.554 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_support_srq": "true", 2023-11-01T04:16:06.554 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_rdma_type": "ib", 2023-11-01T04:16:06.554 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_async_reap_threshold": "5", 2023-11-01T04:16:06.554 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_before_connect": "false", 2023-11-01T04:16:06.554 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_ipv4": "true", 2023-11-01T04:16:06.555 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_ipv6": "false", 2023-11-01T04:16:06.555 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_msgr1": "true", 2023-11-01T04:16:06.555 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_msgr2": "true", 2023-11-01T04:16:06.555 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_port_max": "7568", 2023-11-01T04:16:06.555 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_port_min": "6800", 2023-11-01T04:16:06.555 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_prefer_ipv4": "false", 2023-11-01T04:16:06.556 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_retry_count": "3", 2023-11-01T04:16:06.556 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_bind_retry_delay": "5", 2023-11-01T04:16:06.556 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_blackhole_client": "false", 2023-11-01T04:16:06.556 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_blackhole_mds": "false", 2023-11-01T04:16:06.556 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_blackhole_mgr": "false", 2023-11-01T04:16:06.556 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_blackhole_mon": "false", 2023-11-01T04:16:06.557 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_blackhole_osd": "false", 2023-11-01T04:16:06.557 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_client_mode": "crc secure", 2023-11-01T04:16:06.557 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_cluster_mode": "crc secure", 2023-11-01T04:16:06.557 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_cluster_type": "", 2023-11-01T04:16:06.557 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_compress_secure": "false", 2023-11-01T04:16:06.557 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_connection_idle_timeout": "900", 2023-11-01T04:16:06.558 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_connection_ready_timeout": "10", 2023-11-01T04:16:06.558 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_crc_data": "true", 2023-11-01T04:16:06.558 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_crc_header": "true", 2023-11-01T04:16:06.558 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_die_on_bad_msg": "false", 2023-11-01T04:16:06.558 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_die_on_bug": "true", 2023-11-01T04:16:06.559 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_die_on_old_message": "true", 2023-11-01T04:16:06.559 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_die_on_skipped_message": "false", 2023-11-01T04:16:06.559 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_die_on_unhandled_msg": "false", 2023-11-01T04:16:06.559 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dispatch_throttle_bytes": "104857600", 2023-11-01T04:16:06.560 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_coremask": "0xF", 2023-11-01T04:16:06.560 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_debug_allow_loopback": "false", 2023-11-01T04:16:06.560 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_devs_allowlist": "", 2023-11-01T04:16:06.560 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_enable_tso": "true", 2023-11-01T04:16:06.560 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2023-11-01T04:16:06.561 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_host_ipv4_addr": "", 2023-11-01T04:16:06.561 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_hugepages": "", 2023-11-01T04:16:06.561 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_hw_flow_control": "true", 2023-11-01T04:16:06.561 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2023-11-01T04:16:06.561 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_lro": "true", 2023-11-01T04:16:06.561 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_memory_channel": "4", 2023-11-01T04:16:06.562 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2023-11-01T04:16:06.562 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_pmd": "", 2023-11-01T04:16:06.562 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_port_id": "0", 2023-11-01T04:16:06.562 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2023-11-01T04:16:06.562 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dump_corrupt_message_level": "1", 2023-11-01T04:16:06.562 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_dump_on_send": "false", 2023-11-01T04:16:06.563 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_initial_backoff": "0.200000", 2023-11-01T04:16:06.563 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_inject_delay_max": "1.000000", 2023-11-01T04:16:06.563 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_inject_delay_probability": "0.000000", 2023-11-01T04:16:06.563 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_inject_delay_type": "", 2023-11-01T04:16:06.563 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_inject_internal_delays": "0.000000", 2023-11-01T04:16:06.564 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_inject_network_congestion": "0", 2023-11-01T04:16:06.564 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_inject_socket_failures": "0", 2023-11-01T04:16:06.564 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_learn_addr_from_peer": "true", 2023-11-01T04:16:06.564 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_max_accept_failures": "4", 2023-11-01T04:16:06.564 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_max_backoff": "15.000000", 2023-11-01T04:16:06.564 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_mon_client_mode": "secure crc", 2023-11-01T04:16:06.565 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_mon_cluster_mode": "secure crc", 2023-11-01T04:16:06.565 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_mon_service_mode": "secure crc", 2023-11-01T04:16:06.565 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_osd_compress_min_size": "1024", 2023-11-01T04:16:06.565 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_osd_compress_mode": "none", 2023-11-01T04:16:06.565 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_osd_compression_algorithm": "snappy", 2023-11-01T04:16:06.566 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2023-11-01T04:16:06.566 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_pq_min_cost": "65536", 2023-11-01T04:16:06.566 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_public_type": "", 2023-11-01T04:16:06.566 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_service_mode": "crc secure", 2023-11-01T04:16:06.566 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_tcp_listen_backlog": "512", 2023-11-01T04:16:06.566 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_tcp_nodelay": "true", 2023-11-01T04:16:06.567 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_tcp_prefetch_max_size": "4096", 2023-11-01T04:16:06.567 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_tcp_rcvbuf": "0", 2023-11-01T04:16:06.567 INFO:teuthology.orchestra.run.smithi153.stdout: "ms_type": "async+posix", 2023-11-01T04:16:06.567 INFO:teuthology.orchestra.run.smithi153.stdout: "no_config_file": "false", 2023-11-01T04:16:06.567 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter_completion_locks_per_session": "32", 2023-11-01T04:16:06.567 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter_debug_inject_relock_delay": "false", 2023-11-01T04:16:06.568 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter_inflight_op_bytes": "104857600", 2023-11-01T04:16:06.568 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter_inflight_ops": "1024", 2023-11-01T04:16:06.568 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter_inject_no_watch_ping": "false", 2023-11-01T04:16:06.568 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter_retry_writes_after_first_reply": "false", 2023-11-01T04:16:06.568 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter_tick_interval": "5.000000", 2023-11-01T04:16:06.568 INFO:teuthology.orchestra.run.smithi153.stdout: "objecter_timeout": "10.000000", 2023-11-01T04:16:06.569 INFO:teuthology.orchestra.run.smithi153.stdout: "objectstore_blackhole": "false", 2023-11-01T04:16:06.569 INFO:teuthology.orchestra.run.smithi153.stdout: "openssl_engine_opts": "", 2023-11-01T04:16:06.569 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_agent_delay_time": "5.000000", 2023-11-01T04:16:06.569 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_agent_hist_halflife": "1000", 2023-11-01T04:16:06.569 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_agent_max_low_ops": "2", 2023-11-01T04:16:06.569 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_agent_max_ops": "4", 2023-11-01T04:16:06.570 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_agent_min_evict_effort": "0.100000", 2023-11-01T04:16:06.570 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_agent_quantize_effort": "0.100000", 2023-11-01T04:16:06.570 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_agent_slop": "0.020000", 2023-11-01T04:16:06.570 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_aggregated_slow_ops_logging": "true", 2023-11-01T04:16:06.570 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_allow_recovery_below_min_size": "true", 2023-11-01T04:16:06.571 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_asio_thread_count": "2", 2023-11-01T04:16:06.571 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_async_recovery_min_cost": "100", 2023-11-01T04:16:06.571 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_auto_mark_unfound_lost": "false", 2023-11-01T04:16:06.571 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_backfill_retry_interval": "30.000000", 2023-11-01T04:16:06.571 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_backfill_scan_max": "512", 2023-11-01T04:16:06.571 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_backfill_scan_min": "64", 2023-11-01T04:16:06.572 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_backoff_on_degraded": "false", 2023-11-01T04:16:06.572 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_backoff_on_peering": "false", 2023-11-01T04:16:06.572 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_backoff_on_unfound": "true", 2023-11-01T04:16:06.572 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_beacon_report_interval": "300", 2023-11-01T04:16:06.572 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bench_duration": "30", 2023-11-01T04:16:06.572 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bench_large_size_max_throughput": "104857600", 2023-11-01T04:16:06.573 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bench_max_block_size": "67108864", 2023-11-01T04:16:06.573 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_bench_small_size_max_iops": "100", 2023-11-01T04:16:06.573 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_blkin_trace_all": "false", 2023-11-01T04:16:06.573 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_blocked_scrub_grace_period": "120", 2023-11-01T04:16:06.573 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2023-11-01T04:16:06.574 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2023-11-01T04:16:06.574 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2023-11-01T04:16:06.574 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_check_for_log_corruption": "false", 2023-11-01T04:16:06.574 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_check_max_object_name_len_on_startup": "true", 2023-11-01T04:16:06.574 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-11-01T04:16:06.574 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2023-11-01T04:16:06.575 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-11-01T04:16:06.575 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_class_update_on_start": "true", 2023-11-01T04:16:06.575 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_client_message_cap": "256", 2023-11-01T04:16:06.575 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_client_message_size_cap": "524288000", 2023-11-01T04:16:06.575 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_client_op_priority": "63", 2023-11-01T04:16:06.576 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_client_watch_timeout": "30", 2023-11-01T04:16:06.576 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_command_max_records": "256", 2023-11-01T04:16:06.576 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_compact_on_start": "false", 2023-11-01T04:16:06.576 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_copyfrom_max_chunk": "8388608", 2023-11-01T04:16:06.576 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_crush_chooseleaf_type": "0", 2023-11-01T04:16:06.576 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_crush_initial_weight": "-1.000000", 2023-11-01T04:16:06.577 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_crush_update_on_start": "true", 2023-11-01T04:16:06.577 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_crush_update_weight_set": "true", 2023-11-01T04:16:06.577 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2023-11-01T04:16:06.577 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2023-11-01T04:16:06.577 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2023-11-01T04:16:06.577 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_drop_ping_duration": "0", 2023-11-01T04:16:06.578 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_drop_ping_probability": "0.000000", 2023-11-01T04:16:06.578 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_feed_pullee": "-1", 2023-11-01T04:16:06.578 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_inject_copyfrom_error": "false", 2023-11-01T04:16:06.578 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2023-11-01T04:16:06.578 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_inject_dispatch_delay_probability": "0.100000", 2023-11-01T04:16:06.578 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_misdirected_ops": "false", 2023-11-01T04:16:06.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_no_acting_change": "false", 2023-11-01T04:16:06.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_no_purge_strays": "false", 2023-11-01T04:16:06.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_op_order": "false", 2023-11-01T04:16:06.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_pg_log_writeout": "false", 2023-11-01T04:16:06.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_pretend_recovery_active": "false", 2023-11-01T04:16:06.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_random_push_read_error": "0.000000", 2023-11-01T04:16:06.580 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2023-11-01T04:16:06.580 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_shutdown": "false", 2023-11-01T04:16:06.580 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2023-11-01T04:16:06.580 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2023-11-01T04:16:06.580 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_trim_objects": "false", 2023-11-01T04:16:06.581 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_verify_cached_snaps": "false", 2023-11-01T04:16:06.581 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_verify_missing_on_start": "false", 2023-11-01T04:16:06.581 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_verify_snaps": "false", 2023-11-01T04:16:06.581 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_debug_verify_stray_on_activate": "false", 2023-11-01T04:16:06.581 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_deep_scrub_interval": "604800.000000", 2023-11-01T04:16:06.581 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_deep_scrub_keys": "1024", 2023-11-01T04:16:06.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2023-11-01T04:16:06.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2023-11-01T04:16:06.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2023-11-01T04:16:06.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_deep_scrub_stride": "524288", 2023-11-01T04:16:06.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2023-11-01T04:16:06.583 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_default_data_pool_replay_window": "5", 2023-11-01T04:16:06.583 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_default_notify_timeout": "30", 2023-11-01T04:16:06.583 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_delete_sleep": "0.000000", 2023-11-01T04:16:06.583 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_delete_sleep_hdd": "5.000000", 2023-11-01T04:16:06.583 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_delete_sleep_hybrid": "1.000000", 2023-11-01T04:16:06.583 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_delete_sleep_ssd": "1.000000", 2023-11-01T04:16:06.584 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_discard_disconnected_ops": "true", 2023-11-01T04:16:06.584 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_enable_op_tracker": "true", 2023-11-01T04:16:06.584 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2023-11-01T04:16:06.584 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_extblkdev_plugins": "vdo", 2023-11-01T04:16:06.584 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_failsafe_full_ratio": "0.970000", 2023-11-01T04:16:06.585 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_fast_fail_on_connection_refused": "true", 2023-11-01T04:16:06.585 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_fast_info": "true", 2023-11-01T04:16:06.585 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_fast_shutdown": "true", 2023-11-01T04:16:06.585 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_fast_shutdown_notify_mon": "true", 2023-11-01T04:16:06.585 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_fast_shutdown_timeout": "15", 2023-11-01T04:16:06.585 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_find_best_info_ignore_history_les": "false", 2023-11-01T04:16:06.586 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_force_auth_primary_missing_objects": "100", 2023-11-01T04:16:06.586 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2023-11-01T04:16:06.586 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_function_tracing": "false", 2023-11-01T04:16:06.586 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_heartbeat_grace": "20", 2023-11-01T04:16:06.586 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_heartbeat_interval": "6", 2023-11-01T04:16:06.586 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2023-11-01T04:16:06.587 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_heartbeat_min_peers": "10", 2023-11-01T04:16:06.587 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_heartbeat_min_size": "2000", 2023-11-01T04:16:06.587 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_heartbeat_stale": "600", 2023-11-01T04:16:06.587 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2023-11-01T04:16:06.587 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_hit_set_max_size": "100000", 2023-11-01T04:16:06.587 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_hit_set_min_size": "1000", 2023-11-01T04:16:06.588 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_hit_set_namespace": ".ceph-internal", 2023-11-01T04:16:06.588 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_ignore_stale_divergent_priors": "false", 2023-11-01T04:16:06.588 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2023-11-01T04:16:06.588 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_inject_failure_on_pg_removal": "false", 2023-11-01T04:16:06.588 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2023-11-01T04:16:06.588 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_journal_flush_on_shutdown": "true", 2023-11-01T04:16:06.589 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_journal_size": "5120", 2023-11-01T04:16:06.589 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_kill_backfill_at": "0", 2023-11-01T04:16:06.589 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_loop_before_reset_tphandle": "64", 2023-11-01T04:16:06.589 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_map_cache_size": "50", 2023-11-01T04:16:06.589 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_map_dedup": "true", 2023-11-01T04:16:06.589 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_map_message_max": "40", 2023-11-01T04:16:06.590 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_map_message_max_bytes": "10485760", 2023-11-01T04:16:06.590 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_map_share_max_epochs": "40", 2023-11-01T04:16:06.590 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_attr_name_len": "100", 2023-11-01T04:16:06.590 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_attr_size": "0", 2023-11-01T04:16:06.590 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_backfills": "1", 2023-11-01T04:16:06.591 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_markdown_count": "5", 2023-11-01T04:16:06.591 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_markdown_period": "600", 2023-11-01T04:16:06.591 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_object_name_len": "2048", 2023-11-01T04:16:06.591 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_object_namespace_len": "256", 2023-11-01T04:16:06.591 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_object_size": "134217728", 2023-11-01T04:16:06.591 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2023-11-01T04:16:06.592 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_omap_entries_per_request": "1024", 2023-11-01T04:16:06.592 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_pg_blocked_by": "16", 2023-11-01T04:16:06.592 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_pg_log_entries": "10000", 2023-11-01T04:16:06.592 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2023-11-01T04:16:06.592 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_pgls": "1024", 2023-11-01T04:16:06.593 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_push_cost": "8388608", 2023-11-01T04:16:06.593 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_push_objects": "10", 2023-11-01T04:16:06.593 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_scrubs": "3", 2023-11-01T04:16:06.593 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2023-11-01T04:16:06.593 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_trimming_pgs": "2", 2023-11-01T04:16:06.594 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_write_op_reply_len": "64", 2023-11-01T04:16:06.594 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_max_write_size": "90", 2023-11-01T04:16:06.594 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2023-11-01T04:16:06.594 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2023-11-01T04:16:06.594 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2023-11-01T04:16:06.595 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2023-11-01T04:16:06.595 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2023-11-01T04:16:06.595 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2023-11-01T04:16:06.595 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2023-11-01T04:16:06.595 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_override_recovery_settings": "false", 2023-11-01T04:16:06.596 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_profile": "balanced", 2023-11-01T04:16:06.596 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2023-11-01T04:16:06.596 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2023-11-01T04:16:06.596 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2023-11-01T04:16:06.597 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2023-11-01T04:16:06.597 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2023-11-01T04:16:06.597 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2023-11-01T04:16:06.597 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2023-11-01T04:16:06.597 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2023-11-01T04:16:06.598 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2023-11-01T04:16:06.598 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_scheduler_client_wgt": "1", 2023-11-01T04:16:06.598 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mclock_skip_benchmark": "false", 2023-11-01T04:16:06.598 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_memory_base": "805306368", 2023-11-01T04:16:06.598 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_memory_cache_min": "134217728", 2023-11-01T04:16:06.599 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_memory_cache_resize_interval": "1.000000", 2023-11-01T04:16:06.599 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_memory_expected_fragmentation": "0.150000", 2023-11-01T04:16:06.599 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_memory_target": "4294967296", 2023-11-01T04:16:06.599 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_memory_target_autotune": "false", 2023-11-01T04:16:06.599 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2023-11-01T04:16:06.600 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_min_pg_log_entries": "250", 2023-11-01T04:16:06.600 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_min_recovery_priority": "0", 2023-11-01T04:16:06.600 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mon_heartbeat_interval": "30", 2023-11-01T04:16:06.600 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2023-11-01T04:16:06.600 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mon_report_interval": "5", 2023-11-01T04:16:06.601 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mon_report_max_in_flight": "2", 2023-11-01T04:16:06.601 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_mon_shutdown_timeout": "5.000000", 2023-11-01T04:16:06.601 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_num_cache_shards": "32", 2023-11-01T04:16:06.601 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_num_op_tracker_shard": "32", 2023-11-01T04:16:06.601 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_numa_auto_affinity": "true", 2023-11-01T04:16:06.601 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_numa_node": "-1", 2023-11-01T04:16:06.602 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_numa_prefer_iface": "true", 2023-11-01T04:16:06.602 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2023-11-01T04:16:06.602 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_objecter_finishers": "1", 2023-11-01T04:16:06.602 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_objectstore": "bluestore", 2023-11-01T04:16:06.602 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_objectstore_fuse": "false", 2023-11-01T04:16:06.602 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_objectstore_tracing": "false", 2023-11-01T04:16:06.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_complaint_time": "30.000000", 2023-11-01T04:16:06.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_history_duration": "600", 2023-11-01T04:16:06.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_history_size": "20", 2023-11-01T04:16:06.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_history_slow_op_size": "20", 2023-11-01T04:16:06.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2023-11-01T04:16:06.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_log_threshold": "5", 2023-11-01T04:16:06.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_num_shards": "0", 2023-11-01T04:16:06.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_num_shards_hdd": "5", 2023-11-01T04:16:06.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_num_shards_ssd": "8", 2023-11-01T04:16:06.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_num_threads_per_shard": "0", 2023-11-01T04:16:06.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2023-11-01T04:16:06.604 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2023-11-01T04:16:06.605 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2023-11-01T04:16:06.605 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_pq_min_cost": "65536", 2023-11-01T04:16:06.605 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_queue": "mclock_scheduler", 2023-11-01T04:16:06.605 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_queue_cut_off": "high", 2023-11-01T04:16:06.605 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_thread_suicide_timeout": "150", 2023-11-01T04:16:06.605 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_op_thread_timeout": "15", 2023-11-01T04:16:06.606 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_open_classes_on_start": "true", 2023-11-01T04:16:06.606 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_os_flags": "0", 2023-11-01T04:16:06.606 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_peering_op_priority": "255", 2023-11-01T04:16:06.606 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_delete_cost": "1048576", 2023-11-01T04:16:06.606 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_delete_priority": "5", 2023-11-01T04:16:06.607 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2023-11-01T04:16:06.607 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2023-11-01T04:16:06.607 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_log_dups_tracked": "3000", 2023-11-01T04:16:06.607 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_log_trim_max": "10000", 2023-11-01T04:16:06.607 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_log_trim_min": "100", 2023-11-01T04:16:06.607 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2023-11-01T04:16:06.608 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_object_context_cache_count": "64", 2023-11-01T04:16:06.608 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pg_stat_report_interval_max": "500", 2023-11-01T04:16:06.608 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2023-11-01T04:16:06.608 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_cache_min_evict_age": "0", 2023-11-01T04:16:06.608 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_cache_min_flush_age": "0", 2023-11-01T04:16:06.608 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2023-11-01T04:16:06.609 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2023-11-01T04:16:06.609 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2023-11-01T04:16:06.609 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_crimson": "false", 2023-11-01T04:16:06.609 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_crush_rule": "-1", 2023-11-01T04:16:06.609 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_ec_fast_read": "false", 2023-11-01T04:16:06.609 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2023-11-01T04:16:06.610 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_flag_bulk": "false", 2023-11-01T04:16:06.610 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_flag_hashpspool": "true", 2023-11-01T04:16:06.610 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_flag_nodelete": "false", 2023-11-01T04:16:06.610 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_flag_nopgchange": "false", 2023-11-01T04:16:06.611 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_flag_nosizechange": "false", 2023-11-01T04:16:06.611 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_flags": "0", 2023-11-01T04:16:06.611 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2023-11-01T04:16:06.611 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_min_size": "0", 2023-11-01T04:16:06.611 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2023-11-01T04:16:06.612 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_pg_num": "32", 2023-11-01T04:16:06.612 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_pgp_num": "0", 2023-11-01T04:16:06.612 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2023-11-01T04:16:06.612 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_size": "2", 2023-11-01T04:16:06.612 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_default_type": "replicated", 2023-11-01T04:16:06.612 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2023-11-01T04:16:06.613 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_pool_use_gmt_hitset": "true", 2023-11-01T04:16:06.613 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_push_per_object_cost": "1000", 2023-11-01T04:16:06.613 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_read_ec_check_for_errors": "false", 2023-11-01T04:16:06.613 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recover_clone_overlap": "true", 2023-11-01T04:16:06.613 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recover_clone_overlap_limit": "10", 2023-11-01T04:16:06.613 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_cost": "20971520", 2023-11-01T04:16:06.614 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_delay_start": "0.000000", 2023-11-01T04:16:06.614 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_max_active": "0", 2023-11-01T04:16:06.614 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_max_active_hdd": "3", 2023-11-01T04:16:06.614 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_max_active_ssd": "10", 2023-11-01T04:16:06.614 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_max_chunk": "8388608", 2023-11-01T04:16:06.615 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2023-11-01T04:16:06.615 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_max_single_start": "1", 2023-11-01T04:16:06.615 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_op_priority": "3", 2023-11-01T04:16:06.615 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_op_warn_multiple": "16", 2023-11-01T04:16:06.615 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_priority": "5", 2023-11-01T04:16:06.615 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_retry_interval": "30.000000", 2023-11-01T04:16:06.616 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_sleep": "0.000000", 2023-11-01T04:16:06.616 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_sleep_hdd": "0.100000", 2023-11-01T04:16:06.616 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2023-11-01T04:16:06.616 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_recovery_sleep_ssd": "0.000000", 2023-11-01T04:16:06.616 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_repair_during_recovery": "false", 2023-11-01T04:16:06.617 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_requested_scrub_priority": "120", 2023-11-01T04:16:06.617 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2023-11-01T04:16:06.617 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_rollback_to_cluster_snap": "", 2023-11-01T04:16:06.617 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_auto_repair": "false", 2023-11-01T04:16:06.617 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_auto_repair_num_errors": "5", 2023-11-01T04:16:06.618 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_backoff_ratio": "0.660000", 2023-11-01T04:16:06.618 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_begin_hour": "0", 2023-11-01T04:16:06.618 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_begin_week_day": "0", 2023-11-01T04:16:06.618 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_chunk_max": "25", 2023-11-01T04:16:06.618 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_chunk_min": "5", 2023-11-01T04:16:06.619 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_cost": "52428800", 2023-11-01T04:16:06.619 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_during_recovery": "false", 2023-11-01T04:16:06.619 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_end_hour": "0", 2023-11-01T04:16:06.619 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_end_week_day": "0", 2023-11-01T04:16:06.619 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_event_cost": "4096", 2023-11-01T04:16:06.619 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_extended_sleep": "0.000000", 2023-11-01T04:16:06.620 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2023-11-01T04:16:06.620 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_invalid_stats": "true", 2023-11-01T04:16:06.620 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_load_threshold": "0.500000", 2023-11-01T04:16:06.620 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_max_interval": "604800.000000", 2023-11-01T04:16:06.620 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_max_preemptions": "5", 2023-11-01T04:16:06.621 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_min_interval": "86400.000000", 2023-11-01T04:16:06.621 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_priority": "5", 2023-11-01T04:16:06.621 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_reservation_timeout": "5000", 2023-11-01T04:16:06.621 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_sleep": "0.000000", 2023-11-01T04:16:06.621 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_scrub_slow_reservation_response": "2200", 2023-11-01T04:16:06.622 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_shallow_scrub_chunk_max": "100", 2023-11-01T04:16:06.622 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_shallow_scrub_chunk_min": "50", 2023-11-01T04:16:06.622 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_shutdown_pgref_assert": "false", 2023-11-01T04:16:06.622 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_skip_data_digest": "false", 2023-11-01T04:16:06.622 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_smart_report_timeout": "5", 2023-11-01T04:16:06.622 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_snap_trim_cost": "1048576", 2023-11-01T04:16:06.623 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_snap_trim_priority": "5", 2023-11-01T04:16:06.623 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_snap_trim_sleep": "0.000000", 2023-11-01T04:16:06.623 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2023-11-01T04:16:06.623 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2023-11-01T04:16:06.623 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2023-11-01T04:16:06.623 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_stats_update_period_not_scrubbing": "120", 2023-11-01T04:16:06.624 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_stats_update_period_scrubbing": "15", 2023-11-01T04:16:06.624 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2023-11-01T04:16:06.624 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_target_transaction_size": "30", 2023-11-01T04:16:06.624 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_default_cache_hit_set_count": "4", 2023-11-01T04:16:06.624 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2023-11-01T04:16:06.625 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2023-11-01T04:16:06.625 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2023-11-01T04:16:06.625 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2023-11-01T04:16:06.625 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2023-11-01T04:16:06.625 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2023-11-01T04:16:06.625 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_default_cache_mode": "writeback", 2023-11-01T04:16:06.626 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2023-11-01T04:16:06.626 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tier_promote_max_objects_sec": "25", 2023-11-01T04:16:06.626 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_tracing": "false", 2023-11-01T04:16:06.626 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_use_stale_snap": "false", 2023-11-01T04:16:06.626 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2023-11-01T04:16:06.627 INFO:teuthology.orchestra.run.smithi153.stdout: "osdc_blkin_trace_all": "false", 2023-11-01T04:16:06.627 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_kill_at": "0", 2023-11-01T04:16:06.627 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_max_join_drift": "10", 2023-11-01T04:16:06.627 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_min": "500", 2023-11-01T04:16:06.627 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_min_wait": "0.050000", 2023-11-01T04:16:06.627 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_propose_interval": "1.000000", 2023-11-01T04:16:06.628 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_service_trim_max": "500", 2023-11-01T04:16:06.628 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_service_trim_max_multiplier": "20", 2023-11-01T04:16:06.628 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_service_trim_min": "250", 2023-11-01T04:16:06.628 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_stash_full_interval": "25", 2023-11-01T04:16:06.628 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_trim_max": "500", 2023-11-01T04:16:06.629 INFO:teuthology.orchestra.run.smithi153.stdout: "paxos_trim_min": "250", 2023-11-01T04:16:06.629 INFO:teuthology.orchestra.run.smithi153.stdout: "perf": "true", 2023-11-01T04:16:06.629 INFO:teuthology.orchestra.run.smithi153.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2023-11-01T04:16:06.629 INFO:teuthology.orchestra.run.smithi153.stdout: "plugin_crypto_accelerator": "crypto_isal", 2023-11-01T04:16:06.629 INFO:teuthology.orchestra.run.smithi153.stdout: "plugin_dir": "/usr/lib/ceph", 2023-11-01T04:16:06.630 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "-", 2023-11-01T04:16:06.630 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrv": "", 2023-11-01T04:16:06.630 INFO:teuthology.orchestra.run.smithi153.stdout: "public_bind_addr": "-", 2023-11-01T04:16:06.630 INFO:teuthology.orchestra.run.smithi153.stdout: "public_network": "", 2023-11-01T04:16:06.630 INFO:teuthology.orchestra.run.smithi153.stdout: "public_network_interface": "", 2023-11-01T04:16:06.631 INFO:teuthology.orchestra.run.smithi153.stdout: "qat_compressor_enabled": "false", 2023-11-01T04:16:06.631 INFO:teuthology.orchestra.run.smithi153.stdout: "qat_compressor_session_max_number": "256", 2023-11-01T04:16:06.631 INFO:teuthology.orchestra.run.smithi153.stdout: "rados_mon_op_timeout": "0", 2023-11-01T04:16:06.631 INFO:teuthology.orchestra.run.smithi153.stdout: "rados_osd_op_timeout": "0", 2023-11-01T04:16:06.631 INFO:teuthology.orchestra.run.smithi153.stdout: "rados_tracing": "false", 2023-11-01T04:16:06.632 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_atime_update_interval": "60", 2023-11-01T04:16:06.632 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2023-11-01T04:16:06.632 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_balance_parent_reads": "false", 2023-11-01T04:16:06.632 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_balance_snap_reads": "false", 2023-11-01T04:16:06.632 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_blkin_trace_all": "false", 2023-11-01T04:16:06.632 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_blocklist_expire_seconds": "0", 2023-11-01T04:16:06.633 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_blocklist_on_break_lock": "true", 2023-11-01T04:16:06.633 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache": "true", 2023-11-01T04:16:06.633 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache_block_writes_upfront": "false", 2023-11-01T04:16:06.633 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache_max_dirty": "25165824", 2023-11-01T04:16:06.633 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache_max_dirty_age": "1.000000", 2023-11-01T04:16:06.634 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache_max_dirty_object": "0", 2023-11-01T04:16:06.634 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache_policy": "writearound", 2023-11-01T04:16:06.634 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache_size": "33554432", 2023-11-01T04:16:06.634 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache_target_dirty": "16777216", 2023-11-01T04:16:06.634 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_cache_writethrough_until_flush": "true", 2023-11-01T04:16:06.635 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_clone_copy_on_read": "false", 2023-11-01T04:16:06.635 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_compression_hint": "none", 2023-11-01T04:16:06.635 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_concurrent_management_ops": "10", 2023-11-01T04:16:06.635 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_config_pool_override_update_timestamp": "0", 2023-11-01T04:16:06.635 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_clone_format": "auto", 2023-11-01T04:16:06.635 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_data_pool": "", 2023-11-01T04:16:06.636 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_features": "61", 2023-11-01T04:16:06.636 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_format": "2", 2023-11-01T04:16:06.636 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_map_options": "", 2023-11-01T04:16:06.636 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_order": "22", 2023-11-01T04:16:06.636 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_pool": "rbd", 2023-11-01T04:16:06.636 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2023-11-01T04:16:06.637 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_stripe_count": "0", 2023-11-01T04:16:06.637 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_default_stripe_unit": "0", 2023-11-01T04:16:06.637 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_disable_zero_copy_writes": "true", 2023-11-01T04:16:06.637 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_discard_granularity_bytes": "65536", 2023-11-01T04:16:06.637 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_discard_on_zeroed_write_same": "true", 2023-11-01T04:16:06.637 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_enable_alloc_hint": "true", 2023-11-01T04:16:06.638 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2023-11-01T04:16:06.638 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_io_scheduler": "simple", 2023-11-01T04:16:06.638 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2023-11-01T04:16:06.638 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_commit_age": "5.000000", 2023-11-01T04:16:06.638 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2023-11-01T04:16:06.638 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_max_payload_bytes": "16384", 2023-11-01T04:16:06.639 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_object_flush_age": "0.000000", 2023-11-01T04:16:06.639 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_object_flush_bytes": "1048576", 2023-11-01T04:16:06.639 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_object_flush_interval": "0", 2023-11-01T04:16:06.639 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2023-11-01T04:16:06.639 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2023-11-01T04:16:06.640 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_order": "24", 2023-11-01T04:16:06.640 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_pool": "", 2023-11-01T04:16:06.640 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_journal_splay_width": "4", 2023-11-01T04:16:06.640 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_localize_parent_reads": "false", 2023-11-01T04:16:06.640 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_localize_snap_reads": "false", 2023-11-01T04:16:06.640 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2023-11-01T04:16:06.641 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2023-11-01T04:16:06.641 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2023-11-01T04:16:06.641 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2023-11-01T04:16:06.641 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2023-11-01T04:16:06.641 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2023-11-01T04:16:06.641 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_image_policy_type": "simple", 2023-11-01T04:16:06.642 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2023-11-01T04:16:06.642 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_image_state_check_interval": "30", 2023-11-01T04:16:06.642 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2023-11-01T04:16:06.642 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2023-11-01T04:16:06.642 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2023-11-01T04:16:06.643 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2023-11-01T04:16:06.643 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2023-11-01T04:16:06.643 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_memory_autotune": "true", 2023-11-01T04:16:06.643 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_memory_base": "805306368", 2023-11-01T04:16:06.643 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2023-11-01T04:16:06.643 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_memory_cache_min": "134217728", 2023-11-01T04:16:06.644 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2023-11-01T04:16:06.644 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2023-11-01T04:16:06.644 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_memory_target": "4294967296", 2023-11-01T04:16:06.644 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_perf_stats_prio": "5", 2023-11-01T04:16:06.644 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2023-11-01T04:16:06.645 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2023-11-01T04:16:06.645 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirroring_delete_delay": "0", 2023-11-01T04:16:06.645 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2023-11-01T04:16:06.645 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirroring_replay_delay": "0", 2023-11-01T04:16:06.645 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2023-11-01T04:16:06.646 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2023-11-01T04:16:06.646 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_move_to_trash_on_remove": "false", 2023-11-01T04:16:06.646 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2023-11-01T04:16:06.646 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_mtime_update_interval": "60", 2023-11-01T04:16:06.646 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_non_blocking_aio": "true", 2023-11-01T04:16:06.646 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_op_thread_timeout": "60", 2023-11-01T04:16:06.647 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_op_threads": "1", 2023-11-01T04:16:06.647 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_parent_cache_enabled": "false", 2023-11-01T04:16:06.647 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_persistent_cache_mode": "disabled", 2023-11-01T04:16:06.647 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_persistent_cache_path": "/tmp", 2023-11-01T04:16:06.648 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_persistent_cache_size": "1073741824", 2023-11-01T04:16:06.648 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_plugins": "", 2023-11-01T04:16:06.648 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_bps_burst": "0", 2023-11-01T04:16:06.648 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_bps_burst_seconds": "1", 2023-11-01T04:16:06.648 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_bps_limit": "0", 2023-11-01T04:16:06.648 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_exclude_ops": "0", 2023-11-01T04:16:06.649 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_iops_burst": "0", 2023-11-01T04:16:06.649 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_iops_burst_seconds": "1", 2023-11-01T04:16:06.649 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_iops_limit": "0", 2023-11-01T04:16:06.649 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_read_bps_burst": "0", 2023-11-01T04:16:06.649 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2023-11-01T04:16:06.649 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_read_bps_limit": "0", 2023-11-01T04:16:06.650 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_read_iops_burst": "0", 2023-11-01T04:16:06.650 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2023-11-01T04:16:06.650 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_read_iops_limit": "0", 2023-11-01T04:16:06.650 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_schedule_tick_min": "50", 2023-11-01T04:16:06.651 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_write_bps_burst": "0", 2023-11-01T04:16:06.651 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2023-11-01T04:16:06.651 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_write_bps_limit": "0", 2023-11-01T04:16:06.651 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_write_iops_burst": "0", 2023-11-01T04:16:06.651 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2023-11-01T04:16:06.652 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_qos_write_iops_limit": "0", 2023-11-01T04:16:06.652 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_quiesce_notification_attempts": "10", 2023-11-01T04:16:06.652 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_read_from_replica_policy": "default", 2023-11-01T04:16:06.652 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2023-11-01T04:16:06.652 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_readahead_max_bytes": "524288", 2023-11-01T04:16:06.652 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_readahead_trigger_requests": "10", 2023-11-01T04:16:06.653 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_request_timed_out_seconds": "30", 2023-11-01T04:16:06.653 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_skip_partial_discard": "true", 2023-11-01T04:16:06.653 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2023-11-01T04:16:06.653 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_tracing": "false", 2023-11-01T04:16:06.653 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_validate_names": "true", 2023-11-01T04:16:06.653 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd_validate_pool": "true", 2023-11-01T04:16:06.654 INFO:teuthology.orchestra.run.smithi153.stdout: "restapi_base_url": "", 2023-11-01T04:16:06.654 INFO:teuthology.orchestra.run.smithi153.stdout: "restapi_log_level": "", 2023-11-01T04:16:06.654 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_acl_grants_max_num": "100", 2023-11-01T04:16:06.654 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_admin_entry": "admin", 2023-11-01T04:16:06.654 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2023-11-01T04:16:06.654 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_backend_store": "rados", 2023-11-01T04:16:06.655 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_barbican_url": "", 2023-11-01T04:16:06.655 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_beast_enable_async": "true", 2023-11-01T04:16:06.655 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2023-11-01T04:16:06.655 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_bucket_default_quota_max_size": "-1", 2023-11-01T04:16:06.655 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_bucket_index_max_aio": "128", 2023-11-01T04:16:06.655 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2023-11-01T04:16:06.656 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_bucket_quota_cache_size": "10000", 2023-11-01T04:16:06.656 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_bucket_quota_ttl": "600", 2023-11-01T04:16:06.656 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_bucket_sync_spawn_window": "20", 2023-11-01T04:16:06.656 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_cache_enabled": "true", 2023-11-01T04:16:06.656 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_cache_expiry_interval": "900", 2023-11-01T04:16:06.657 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_cache_lru_size": "10000", 2023-11-01T04:16:06.657 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_config_store": "rados", 2023-11-01T04:16:06.657 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_content_length_compat": "false", 2023-11-01T04:16:06.657 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_copy_obj_progress": "true", 2023-11-01T04:16:06.657 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2023-11-01T04:16:06.657 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_cors_rules_max_num": "100", 2023-11-01T04:16:06.658 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_cross_domain_policy": "", 2023-11-01T04:16:06.658 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_default_encryption_key": "", 2023-11-01T04:16:06.658 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_kmip_addr": "", 2023-11-01T04:16:06.658 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_kmip_ca_path": "", 2023-11-01T04:16:06.658 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_kmip_client_cert": "", 2023-11-01T04:16:06.658 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_kmip_client_key": "", 2023-11-01T04:16:06.659 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_kmip_kms_key_template": "", 2023-11-01T04:16:06.659 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_kmip_password": "", 2023-11-01T04:16:06.659 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2023-11-01T04:16:06.659 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_kmip_username": "", 2023-11-01T04:16:06.659 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_require_ssl": "true", 2023-11-01T04:16:06.659 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2023-11-01T04:16:06.660 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2023-11-01T04:16:06.660 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_backend": "vault", 2023-11-01T04:16:06.660 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2023-11-01T04:16:06.660 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2023-11-01T04:16:06.660 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2023-11-01T04:16:06.660 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2023-11-01T04:16:06.661 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2023-11-01T04:16:06.661 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2023-11-01T04:16:06.661 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2023-11-01T04:16:06.661 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2023-11-01T04:16:06.661 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2023-11-01T04:16:06.661 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2023-11-01T04:16:06.662 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2023-11-01T04:16:06.662 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_suppress_logs": "true", 2023-11-01T04:16:06.662 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_addr": "", 2023-11-01T04:16:06.662 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_auth": "token", 2023-11-01T04:16:06.662 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_namespace": "", 2023-11-01T04:16:06.663 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_prefix": "", 2023-11-01T04:16:06.663 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_secret_engine": "transit", 2023-11-01T04:16:06.663 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_ssl_cacert": "", 2023-11-01T04:16:06.663 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2023-11-01T04:16:06.663 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2023-11-01T04:16:06.663 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_token_file": "", 2023-11-01T04:16:06.664 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_crypt_vault_verify_ssl": "true", 2023-11-01T04:16:06.664 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_curl_buffersize": "524288", 2023-11-01T04:16:06.664 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_curl_low_speed_limit": "1024", 2023-11-01T04:16:06.664 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_curl_low_speed_time": "300", 2023-11-01T04:16:06.664 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_curl_tcp_keepalive": "0", 2023-11-01T04:16:06.664 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_curl_wait_timeout_ms": "1000", 2023-11-01T04:16:06.665 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2023-11-01T04:16:06.665 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2023-11-01T04:16:06.665 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2023-11-01T04:16:06.665 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2023-11-01T04:16:06.665 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d3n_l1_fadvise": "4", 2023-11-01T04:16:06.665 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2023-11-01T04:16:06.666 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d3n_libaio_aio_num": "64", 2023-11-01T04:16:06.666 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d3n_libaio_aio_threads": "20", 2023-11-01T04:16:06.666 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d4n_host": "127.0.0.1", 2023-11-01T04:16:06.666 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_d4n_port": "6379", 2023-11-01T04:16:06.666 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2023-11-01T04:16:06.666 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_data_log_changes_size": "1000", 2023-11-01T04:16:06.667 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_data_log_num_shards": "128", 2023-11-01T04:16:06.667 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_data_log_obj_prefix": "data_log", 2023-11-01T04:16:06.667 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_data_log_window": "30", 2023-11-01T04:16:06.667 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_data_notify_interval_msec": "0", 2023-11-01T04:16:06.667 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_data_sync_poll_interval": "20", 2023-11-01T04:16:06.667 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_data_sync_spawn_window": "20", 2023-11-01T04:16:06.668 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2023-11-01T04:16:06.668 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_debug_inject_set_olh_err": "0", 2023-11-01T04:16:06.668 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_default_data_log_backing": "fifo", 2023-11-01T04:16:06.668 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_default_realm_info_oid": "default.realm", 2023-11-01T04:16:06.668 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_default_region_info_oid": "default.region", 2023-11-01T04:16:06.669 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_default_zone_info_oid": "default.zone", 2023-11-01T04:16:06.669 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2023-11-01T04:16:06.669 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_defer_to_bucket_acls": "", 2023-11-01T04:16:06.669 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_delete_multi_obj_max_num": "1000", 2023-11-01T04:16:06.669 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_admin_lim": "0.000000", 2023-11-01T04:16:06.670 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_admin_res": "100.000000", 2023-11-01T04:16:06.670 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2023-11-01T04:16:06.670 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_auth_lim": "0.000000", 2023-11-01T04:16:06.670 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_auth_res": "200.000000", 2023-11-01T04:16:06.670 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2023-11-01T04:16:06.671 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_data_lim": "0.000000", 2023-11-01T04:16:06.671 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_data_res": "500.000000", 2023-11-01T04:16:06.671 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_data_wgt": "500.000000", 2023-11-01T04:16:06.671 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2023-11-01T04:16:06.671 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_metadata_res": "500.000000", 2023-11-01T04:16:06.672 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2023-11-01T04:16:06.672 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dns_name": "", 2023-11-01T04:16:06.672 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dns_s3website_name": "", 2023-11-01T04:16:06.672 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_dynamic_resharding": "true", 2023-11-01T04:16:06.672 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2023-11-01T04:16:06.673 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_enable_gc_threads": "true", 2023-11-01T04:16:06.673 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_enable_lc_threads": "true", 2023-11-01T04:16:06.673 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_enable_ops_log": "false", 2023-11-01T04:16:06.673 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_enable_quota_threads": "true", 2023-11-01T04:16:06.673 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_enable_static_website": "false", 2023-11-01T04:16:06.673 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_enable_usage_log": "false", 2023-11-01T04:16:06.674 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_enforce_swift_acls": "true", 2023-11-01T04:16:06.674 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_exit_timeout_secs": "120", 2023-11-01T04:16:06.674 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_expose_bucket": "false", 2023-11-01T04:16:06.674 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_extended_http_attrs": "", 2023-11-01T04:16:06.674 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_filter": "none", 2023-11-01T04:16:06.675 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2023-11-01T04:16:06.675 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_frontends": "beast port=7480", 2023-11-01T04:16:06.675 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_max_concurrent_io": "10", 2023-11-01T04:16:06.675 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_max_deferred": "50", 2023-11-01T04:16:06.675 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2023-11-01T04:16:06.675 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_max_objs": "32", 2023-11-01T04:16:06.676 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_max_queue_size": "134213632", 2023-11-01T04:16:06.676 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_max_trim_chunk": "16", 2023-11-01T04:16:06.676 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_obj_min_wait": "7200", 2023-11-01T04:16:06.676 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_processor_max_time": "3600", 2023-11-01T04:16:06.676 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_gc_processor_period": "3600", 2023-11-01T04:16:06.676 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_get_obj_max_req_size": "4194304", 2023-11-01T04:16:06.677 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_get_obj_window_size": "16777216", 2023-11-01T04:16:06.677 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_healthcheck_disabling_path": "", 2023-11-01T04:16:06.677 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ignore_get_invalid_range": "false", 2023-11-01T04:16:06.677 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_init_timeout": "300", 2023-11-01T04:16:06.677 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2023-11-01T04:16:06.678 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2023-11-01T04:16:06.678 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_accepted_admin_roles": "", 2023-11-01T04:16:06.678 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_accepted_reader_roles": "", 2023-11-01T04:16:06.678 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2023-11-01T04:16:06.678 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_admin_domain": "", 2023-11-01T04:16:06.678 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_admin_password": "", 2023-11-01T04:16:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_admin_password_path": "", 2023-11-01T04:16:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_admin_project": "", 2023-11-01T04:16:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_admin_tenant": "", 2023-11-01T04:16:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_admin_token": "", 2023-11-01T04:16:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_admin_token_path": "", 2023-11-01T04:16:06.679 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_admin_user": "", 2023-11-01T04:16:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_api_version": "2", 2023-11-01T04:16:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_barbican_domain": "", 2023-11-01T04:16:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_barbican_password": "", 2023-11-01T04:16:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_barbican_project": "", 2023-11-01T04:16:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_barbican_tenant": "", 2023-11-01T04:16:06.680 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_barbican_user": "", 2023-11-01T04:16:06.681 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2023-11-01T04:16:06.681 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_implicit_tenants": "false", 2023-11-01T04:16:06.681 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2023-11-01T04:16:06.681 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_service_token_enabled": "false", 2023-11-01T04:16:06.681 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_token_cache_size": "10000", 2023-11-01T04:16:06.681 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_url": "", 2023-11-01T04:16:06.682 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_keystone_verify_ssl": "true", 2023-11-01T04:16:06.682 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lc_debug_interval": "-1", 2023-11-01T04:16:06.682 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lc_lock_max_time": "90", 2023-11-01T04:16:06.682 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lc_max_objs": "32", 2023-11-01T04:16:06.682 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lc_max_rules": "1000", 2023-11-01T04:16:06.683 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lc_max_worker": "3", 2023-11-01T04:16:06.683 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lc_max_wp_worker": "3", 2023-11-01T04:16:06.683 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lc_thread_delay": "0", 2023-11-01T04:16:06.683 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2023-11-01T04:16:06.683 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ldap_dnattr": "uid", 2023-11-01T04:16:06.683 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2023-11-01T04:16:06.684 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ldap_searchfilter": "", 2023-11-01T04:16:06.684 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2023-11-01T04:16:06.684 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ldap_uri": "ldaps://", 2023-11-01T04:16:06.684 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2023-11-01T04:16:06.684 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_list_bucket_min_readahead": "1000", 2023-11-01T04:16:06.684 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_list_buckets_max_chunk": "1000", 2023-11-01T04:16:06.685 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_log_http_headers": "", 2023-11-01T04:16:06.685 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_log_nonexistent_bucket": "false", 2023-11-01T04:16:06.685 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2023-11-01T04:16:06.685 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_log_object_name_utc": "false", 2023-11-01T04:16:06.685 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_lua_max_memory_per_state": "128000", 2023-11-01T04:16:06.685 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2023-11-01T04:16:06.686 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_attr_name_len": "0", 2023-11-01T04:16:06.686 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_attr_size": "0", 2023-11-01T04:16:06.686 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_attrs_num_in_req": "0", 2023-11-01T04:16:06.686 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_chunk_size": "4194304", 2023-11-01T04:16:06.686 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_concurrent_requests": "1024", 2023-11-01T04:16:06.687 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2023-11-01T04:16:06.687 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_dynamic_shards": "1999", 2023-11-01T04:16:06.687 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_listing_results": "1000", 2023-11-01T04:16:06.687 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_notify_retries": "10", 2023-11-01T04:16:06.687 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_objs_per_shard": "100000", 2023-11-01T04:16:06.687 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_put_param_size": "1048576", 2023-11-01T04:16:06.688 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_put_size": "5368709120", 2023-11-01T04:16:06.688 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_max_slo_entries": "1000", 2023-11-01T04:16:06.688 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_md_log_max_shards": "64", 2023-11-01T04:16:06.688 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_md_notify_interval_msec": "200", 2023-11-01T04:16:06.688 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_meta_sync_poll_interval": "20", 2023-11-01T04:16:06.688 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_meta_sync_spawn_window": "20", 2023-11-01T04:16:06.689 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_mime_types_file": "/etc/mime.types", 2023-11-01T04:16:06.689 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_mp_lock_max_time": "600", 2023-11-01T04:16:06.689 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_multi_obj_del_max_aio": "16", 2023-11-01T04:16:06.689 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_multipart_min_part_size": "5242880", 2023-11-01T04:16:06.689 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_multipart_part_upload_limit": "10000", 2023-11-01T04:16:06.690 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_fhcache_partitions": "3", 2023-11-01T04:16:06.690 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_fhcache_size": "2017", 2023-11-01T04:16:06.690 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_frontends": "rgw-nfs", 2023-11-01T04:16:06.690 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2023-11-01T04:16:06.690 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_lru_lanes": "5", 2023-11-01T04:16:06.691 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_max_gc": "300", 2023-11-01T04:16:06.691 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_namespace_expire_secs": "300", 2023-11-01T04:16:06.691 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_run_gc_threads": "false", 2023-11-01T04:16:06.691 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_run_lc_threads": "false", 2023-11-01T04:16:06.692 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_run_quota_threads": "false", 2023-11-01T04:16:06.692 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_run_sync_thread": "false", 2023-11-01T04:16:06.692 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_s3_fast_attrs": "false", 2023-11-01T04:16:06.692 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_nfs_write_completion_interval_s": "10", 2023-11-01T04:16:06.692 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_num_async_rados_threads": "32", 2023-11-01T04:16:06.692 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_num_control_oids": "8", 2023-11-01T04:16:06.693 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_numa_node": "-1", 2023-11-01T04:16:06.693 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_obj_stripe_size": "4194304", 2023-11-01T04:16:06.693 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_obj_tombstone_cache_size": "1000", 2023-11-01T04:16:06.693 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_objexp_chunk_size": "100", 2023-11-01T04:16:06.693 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_objexp_gc_interval": "600", 2023-11-01T04:16:06.693 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_objexp_hints_num_shards": "127", 2023-11-01T04:16:06.694 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_olh_pending_timeout_sec": "3600", 2023-11-01T04:16:06.694 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_op_thread_suicide_timeout": "0", 2023-11-01T04:16:06.694 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_op_thread_timeout": "600", 2023-11-01T04:16:06.694 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_op_tracing": "false", 2023-11-01T04:16:06.694 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_opa_token": "", 2023-11-01T04:16:06.695 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_opa_url": "", 2023-11-01T04:16:06.695 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_opa_verify_ssl": "true", 2023-11-01T04:16:06.695 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ops_log_data_backlog": "5242880", 2023-11-01T04:16:06.695 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2023-11-01T04:16:06.695 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ops_log_rados": "false", 2023-11-01T04:16:06.695 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_ops_log_socket_path": "", 2023-11-01T04:16:06.696 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_override_bucket_index_max_shards": "0", 2023-11-01T04:16:06.696 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2023-11-01T04:16:06.696 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2023-11-01T04:16:06.696 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_period_push_interval": "2.000000", 2023-11-01T04:16:06.696 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_period_push_interval_max": "30.000000", 2023-11-01T04:16:06.696 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_period_root_pool": ".rgw.root", 2023-11-01T04:16:06.697 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_policy_reject_invalid_principals": "true", 2023-11-01T04:16:06.697 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2023-11-01T04:16:06.697 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_posix_cache_lanes": "3", 2023-11-01T04:16:06.697 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_posix_cache_lmdb_count": "3", 2023-11-01T04:16:06.697 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_posix_cache_max_buckets": "100", 2023-11-01T04:16:06.697 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_posix_cache_partitions": "3", 2023-11-01T04:16:06.698 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2023-11-01T04:16:06.698 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_print_continue": "true", 2023-11-01T04:16:06.698 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_print_prohibited_content_length": "false", 2023-11-01T04:16:06.698 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_put_obj_max_window_size": "67108864", 2023-11-01T04:16:06.698 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_put_obj_min_window_size": "16777216", 2023-11-01T04:16:06.698 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2023-11-01T04:16:06.699 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_rados_pool_recovery_priority": "5", 2023-11-01T04:16:06.699 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_rados_tracing": "false", 2023-11-01T04:16:06.699 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_realm": "", 2023-11-01T04:16:06.699 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_realm_id": "", 2023-11-01T04:16:06.699 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_realm_root_pool": ".rgw.root", 2023-11-01T04:16:06.699 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_region": "", 2023-11-01T04:16:06.700 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_region_root_pool": ".rgw.root", 2023-11-01T04:16:06.700 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_relaxed_region_enforcement": "false", 2023-11-01T04:16:06.700 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_relaxed_s3_bucket_names": "false", 2023-11-01T04:16:06.700 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2023-11-01T04:16:06.700 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_request_uri": "", 2023-11-01T04:16:06.700 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_reshard_batch_size": "64", 2023-11-01T04:16:06.701 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_reshard_bucket_lock_duration": "360", 2023-11-01T04:16:06.701 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_reshard_max_aio": "128", 2023-11-01T04:16:06.701 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_reshard_num_logs": "16", 2023-11-01T04:16:06.701 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_reshard_thread_interval": "600", 2023-11-01T04:16:06.701 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_resolve_cname": "false", 2023-11-01T04:16:06.702 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_rest_getusage_op_compat": "false", 2023-11-01T04:16:06.702 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_run_sync_thread": "true", 2023-11-01T04:16:06.702 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_s3_auth_order": "sts, external, local", 2023-11-01T04:16:06.702 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_s3_auth_use_keystone": "false", 2023-11-01T04:16:06.702 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_s3_auth_use_ldap": "false", 2023-11-01T04:16:06.702 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_s3_auth_use_rados": "true", 2023-11-01T04:16:06.703 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_s3_auth_use_sts": "false", 2023-11-01T04:16:06.703 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_s3_client_max_sig_ver": "-1", 2023-11-01T04:16:06.703 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_s3_success_create_obj_status": "0", 2023-11-01T04:16:06.703 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_safe_max_objects_per_shard": "102400", 2023-11-01T04:16:06.703 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_scheduler_type": "throttler", 2023-11-01T04:16:06.704 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_script_uri": "", 2023-11-01T04:16:06.704 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_service_provider_name": "", 2023-11-01T04:16:06.704 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_shard_warning_threshold": "90.000000", 2023-11-01T04:16:06.704 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sts_client_id": "", 2023-11-01T04:16:06.704 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sts_client_secret": "", 2023-11-01T04:16:06.704 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sts_entry": "sts", 2023-11-01T04:16:06.705 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sts_key": "sts", 2023-11-01T04:16:06.705 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sts_max_session_duration": "43200", 2023-11-01T04:16:06.705 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sts_min_session_duration": "900", 2023-11-01T04:16:06.705 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sts_token_introspection_url": "", 2023-11-01T04:16:06.705 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_account_in_url": "false", 2023-11-01T04:16:06.706 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_auth_entry": "auth", 2023-11-01T04:16:06.706 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_auth_url": "", 2023-11-01T04:16:06.706 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_custom_header": "", 2023-11-01T04:16:06.706 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_enforce_content_length": "false", 2023-11-01T04:16:06.706 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_need_stats": "true", 2023-11-01T04:16:06.706 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_tenant_name": "", 2023-11-01T04:16:06.707 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_token_expiration": "86400", 2023-11-01T04:16:06.707 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_url": "", 2023-11-01T04:16:06.707 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_url_prefix": "swift", 2023-11-01T04:16:06.707 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_swift_versioning_enabled": "false", 2023-11-01T04:16:06.707 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2023-11-01T04:16:06.708 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2023-11-01T04:16:06.708 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_lease_period": "120", 2023-11-01T04:16:06.708 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2023-11-01T04:16:06.708 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_log_trim_interval": "1200", 2023-11-01T04:16:06.708 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_log_trim_max_buckets": "16", 2023-11-01T04:16:06.708 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2023-11-01T04:16:06.709 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2023-11-01T04:16:06.709 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_obj_etag_verify": "false", 2023-11-01T04:16:06.709 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_trace_history_size": "4096", 2023-11-01T04:16:06.709 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_trace_per_node_log_size": "32", 2023-11-01T04:16:06.709 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2023-11-01T04:16:06.709 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_thread_pool_size": "512", 2023-11-01T04:16:06.710 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_topic_persistency_max_retries": "0", 2023-11-01T04:16:06.710 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_topic_persistency_sleep_duration": "0", 2023-11-01T04:16:06.710 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_topic_persistency_time_to_live": "0", 2023-11-01T04:16:06.710 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_torrent_comment": "", 2023-11-01T04:16:06.710 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_torrent_createby": "", 2023-11-01T04:16:06.711 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_torrent_encoding": "", 2023-11-01T04:16:06.711 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_torrent_flag": "false", 2023-11-01T04:16:06.711 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_torrent_max_size": "5368709120", 2023-11-01T04:16:06.711 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_torrent_origin": "", 2023-11-01T04:16:06.711 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_torrent_sha_unit": "524288", 2023-11-01T04:16:06.711 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_torrent_tracker": "", 2023-11-01T04:16:06.712 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_trust_forwarded_https": "false", 2023-11-01T04:16:06.712 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_usage_log_flush_threshold": "1024", 2023-11-01T04:16:06.712 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_usage_log_tick_interval": "30", 2023-11-01T04:16:06.712 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_usage_max_shards": "32", 2023-11-01T04:16:06.712 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_usage_max_user_shards": "1", 2023-11-01T04:16:06.712 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_use_opa_authz": "false", 2023-11-01T04:16:06.713 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_default_quota_max_objects": "-1", 2023-11-01T04:16:06.713 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_default_quota_max_size": "-1", 2023-11-01T04:16:06.713 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_max_buckets": "1000", 2023-11-01T04:16:06.713 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_policies_max_num": "100", 2023-11-01T04:16:06.713 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2023-11-01T04:16:06.713 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_quota_sync_idle_users": "false", 2023-11-01T04:16:06.714 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_quota_sync_interval": "86400", 2023-11-01T04:16:06.714 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_quota_sync_wait_time": "86400", 2023-11-01T04:16:06.714 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_user_unique_email": "true", 2023-11-01T04:16:06.714 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_verify_ssl": "true", 2023-11-01T04:16:06.714 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_website_routing_rules_max_num": "50", 2023-11-01T04:16:06.715 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_zone": "", 2023-11-01T04:16:06.715 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_zone_id": "", 2023-11-01T04:16:06.715 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_zone_root_pool": ".rgw.root", 2023-11-01T04:16:06.715 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_zonegroup": "", 2023-11-01T04:16:06.715 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_zonegroup_id": "", 2023-11-01T04:16:06.715 INFO:teuthology.orchestra.run.smithi153.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2023-11-01T04:16:06.716 INFO:teuthology.orchestra.run.smithi153.stdout: "rgwlc_auto_session_clear": "true", 2023-11-01T04:16:06.716 INFO:teuthology.orchestra.run.smithi153.stdout: "rgwlc_skip_bucket_step": "false", 2023-11-01T04:16:06.716 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_block_size": "4096", 2023-11-01T04:16:06.716 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_bloom_bits_per_key": "20", 2023-11-01T04:16:06.716 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2023-11-01T04:16:06.716 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2023-11-01T04:16:06.717 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cache_row_ratio": "0.000000", 2023-11-01T04:16:06.717 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cache_shard_bits": "4", 2023-11-01T04:16:06.717 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cache_size": "536870912", 2023-11-01T04:16:06.717 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cache_type": "binned_lru", 2023-11-01T04:16:06.717 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cf_compact_on_deletion": "true", 2023-11-01T04:16:06.717 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2023-11-01T04:16:06.718 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2023-11-01T04:16:06.718 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_collect_compaction_stats": "false", 2023-11-01T04:16:06.718 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_collect_extended_stats": "false", 2023-11-01T04:16:06.718 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_collect_memory_stats": "false", 2023-11-01T04:16:06.718 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_delete_range_threshold": "1048576", 2023-11-01T04:16:06.718 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_index_type": "binary_search", 2023-11-01T04:16:06.719 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_log_to_ceph_log": "true", 2023-11-01T04:16:06.719 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_metadata_block_size": "4096", 2023-11-01T04:16:06.719 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_partition_filters": "false", 2023-11-01T04:16:06.719 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_perf": "false", 2023-11-01T04:16:06.719 INFO:teuthology.orchestra.run.smithi153.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2023-11-01T04:16:06.720 INFO:teuthology.orchestra.run.smithi153.stdout: "rotating_keys_bootstrap_timeout": "30", 2023-11-01T04:16:06.720 INFO:teuthology.orchestra.run.smithi153.stdout: "rotating_keys_renewal_timeout": "10", 2023-11-01T04:16:06.720 INFO:teuthology.orchestra.run.smithi153.stdout: "run_dir": "/var/run/ceph", 2023-11-01T04:16:06.720 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_block_create": "true", 2023-11-01T04:16:06.720 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_cache_lru_size": "67108864", 2023-11-01T04:16:06.720 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_cbjournal_size": "5368709120", 2023-11-01T04:16:06.721 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_default_max_object_size": "16777216", 2023-11-01T04:16:06.721 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_default_object_metadata_reservation": "16777216", 2023-11-01T04:16:06.721 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_device_size": "53687091200", 2023-11-01T04:16:06.721 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_journal_batch_capacity": "16", 2023-11-01T04:16:06.722 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_journal_batch_flush_size": "16777216", 2023-11-01T04:16:06.722 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2023-11-01T04:16:06.722 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_journal_iodepth_limit": "5", 2023-11-01T04:16:06.722 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_main_device_type": "SSD", 2023-11-01T04:16:06.722 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_max_concurrent_transactions": "8", 2023-11-01T04:16:06.722 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2023-11-01T04:16:06.723 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2023-11-01T04:16:06.723 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2023-11-01T04:16:06.723 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_obj_data_write_amplification": "1.250000", 2023-11-01T04:16:06.723 INFO:teuthology.orchestra.run.smithi153.stdout: "seastore_segment_size": "67108864", 2023-11-01T04:16:06.723 INFO:teuthology.orchestra.run.smithi153.stdout: "set_keepcaps": "false", 2023-11-01T04:16:06.723 INFO:teuthology.orchestra.run.smithi153.stdout: "setgroup": "", 2023-11-01T04:16:06.724 INFO:teuthology.orchestra.run.smithi153.stdout: "setuser": "", 2023-11-01T04:16:06.724 INFO:teuthology.orchestra.run.smithi153.stdout: "setuser_match_path": "", 2023-11-01T04:16:06.724 INFO:teuthology.orchestra.run.smithi153.stdout: "target_max_misplaced_ratio": "0.050000", 2023-11-01T04:16:06.724 INFO:teuthology.orchestra.run.smithi153.stdout: "thp": "false", 2023-11-01T04:16:06.724 INFO:teuthology.orchestra.run.smithi153.stdout: "threadpool_default_timeout": "60", 2023-11-01T04:16:06.724 INFO:teuthology.orchestra.run.smithi153.stdout: "threadpool_empty_queue_max_wait": "2", 2023-11-01T04:16:06.725 INFO:teuthology.orchestra.run.smithi153.stdout: "throttler_perf_counter": "true" 2023-11-01T04:16:06.725 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:06.726 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2023-11-01T04:16:06.871 INFO:teuthology.orchestra.run.smithi153.stdout:mon.a: {} 2023-11-01T04:16:06.871 INFO:teuthology.orchestra.run.smithi153.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2023-11-01T04:16:06.883 INFO:teuthology.orchestra.run.smithi153.stdout:mon.b: {} 2023-11-01T04:16:06.884 INFO:teuthology.orchestra.run.smithi153.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2023-11-01T04:16:06.896 INFO:teuthology.orchestra.run.smithi153.stdout:mon.c: {} 2023-11-01T04:16:06.896 INFO:teuthology.orchestra.run.smithi153.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2023-11-01T04:16:06.912 DEBUG:teuthology.orchestra.run.smithi173:> type ceph-objectstore-tool 2023-11-01T04:16:06.918 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-11-01T04:16:06.918 DEBUG:teuthology.orchestra.run.smithi190:> type ceph-objectstore-tool 2023-11-01T04:16:06.924 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-11-01T04:16:06.924 DEBUG:teuthology.orchestra.run.smithi181:> type ceph-objectstore-tool 2023-11-01T04:16:06.930 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-11-01T04:16:06.931 DEBUG:teuthology.orchestra.run.smithi153:> type ceph-objectstore-tool 2023-11-01T04:16:06.935 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-11-01T04:16:06.935 INFO:teuthology.run_tasks:Running task rados... 2023-11-01T04:16:06.946 INFO:tasks.rados:Beginning rados... 2023-11-01T04:16:06.946 DEBUG:teuthology.run_tasks:Unwinding manager rados 2023-11-01T04:16:06.968 INFO:tasks.rados:joining rados 2023-11-01T04:16:06.968 INFO:tasks.thrashosds.thrasher:starting do_thrash 2023-11-01T04:16:06.968 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2023-11-01T04:16:06.968 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-01T04:16:06.969 INFO:tasks.thrashosds.thrasher:inject_pause on osd.9 2023-11-01T04:16:06.969 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-11-01T04:16:06.969 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-11-01T04:16:06.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok config set filestore_inject_stall 3 2023-11-01T04:16:06.970 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2023-11-01T04:16:06.970 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2023-11-01T04:16:06.971 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-01T04:16:06.971 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2023-11-01T04:16:06.971 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-01T04:16:06.972 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2023-11-01T04:16:06.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-11-01T04:16:06.973 INFO:tasks.rados:clients are ['client.0'] 2023-11-01T04:16:06.973 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set shecprofile c=2 crush-failure-domain=osd k=4 m=3 name=shecprofile plugin=shec 2023-11-01T04:16:06.980 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:06.977+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:07.072 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:07.072+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:07.102 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:07.102 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:16:07.102 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2023-11-01T04:16:07.103 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:07.103 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:07.103 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-11-01T04:16:07.104 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:07.118 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:07.119 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-11-01T04:16:07.120 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-01T04:16:07.137 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:07.137 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-01T04:16:07.172 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:07.172 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-01T04:16:07.173 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:07.172+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:07.206 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:07.206 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-01T04:16:07.238 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:07.238 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-01T04:16:07.271 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:07.271 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-01T04:16:07.273 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:07.273 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:07.274 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:07.272+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:07.286 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-01T04:16:07.291 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:07.291 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-01T04:16:07.311 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:07.311 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-01T04:16:07.327 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:07.327 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-01T04:16:07.347 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:07.347 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-01T04:16:07.366 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:07.366 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-01T04:16:07.375 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:07.373+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:07.385 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:07.386 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-01T04:16:07.398 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:07.399 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:07.401 INFO:teuthology.orchestra.run.smithi153.stdout:osd.12: {} 2023-11-01T04:16:07.401 INFO:teuthology.orchestra.run.smithi153.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-01T04:16:07.412 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-01T04:16:07.421 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:07.422 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-01T04:16:07.441 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:07.441 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-01T04:16:07.460 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:07.460 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-01T04:16:07.476 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:07.473+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:07.526 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:07.526 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:07.539 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-01T04:16:07.587 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:07.585+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:07.645 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:07.645 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:07.657 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-01T04:16:07.677 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:07.676+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:07.778 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:07.773+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:07.816 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:07.817 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:07.830 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-01T04:16:07.849 INFO:teuthology.orchestra.run.smithi153.stderr:load: shec 2023-11-01T04:16:07.850 INFO:teuthology.orchestra.run.smithi153.stderr:noscrub is set 2023-11-01T04:16:07.873 INFO:tasks.rados:starting run 0 out of 1 2023-11-01T04:16:07.873 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2023-11-01T04:16:07.873 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure shecprofile 2023-11-01T04:16:07.879 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:07.880+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:07.967 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:07.968 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:07.981 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:07.978+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:07.982 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-01T04:16:08.082 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:08.080+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:08.094 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:08.094 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:08.108 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-01T04:16:08.183 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:08.177+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:08.218 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:08.218 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:08.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-01T04:16:08.319 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:08.317+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:08.384 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:08.384+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:08.385 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:08.386 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:08.399 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-01T04:16:08.485 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:08.484+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:08.562 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:08.562 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:08.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-01T04:16:08.623 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:08.621+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:08.687 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:08.682+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:08.696 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:08.696 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:08.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-01T04:16:08.787 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:08.785+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:08.842 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:08.842 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:08.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-01T04:16:08.888 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:08.888+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:08.977 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:08.978 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:08.989 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:08.986+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:08.992 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-01T04:16:09.090 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:09.088+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:09.154 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:09.155 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:09.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-01T04:16:09.217 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:09.214+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:09.289 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:09.289 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:09.291 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:09.288+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:09.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-01T04:16:09.392 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:09.392+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:09.470 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:09.470 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:09.477 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-01T04:16:09.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-01T04:16:09.493 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:09.492+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:09.594 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:09.589+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:09.605 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:09.605 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:09.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-01T04:16:09.627 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:09.627 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-11-01T04:16:09.645 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:09.645 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-01T04:16:09.665 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:09.665 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-01T04:16:09.685 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:09.685 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-01T04:16:09.695 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:09.692+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:09.701 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:09.702 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-01T04:16:09.720 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:09.720 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-01T04:16:09.741 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:09.741 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-01T04:16:09.742 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:09.742 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:09.756 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-01T04:16:09.761 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:09.761 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-01T04:16:09.778 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:09.779 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-01T04:16:09.799 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:09.799 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-01T04:16:09.801 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:09.798+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:09.819 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:09.819 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-01T04:16:09.840 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:09.841 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-01T04:16:09.857 INFO:teuthology.orchestra.run.smithi153.stdout:osd.12: {} 2023-11-01T04:16:09.858 INFO:teuthology.orchestra.run.smithi153.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-01T04:16:09.866 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_0' created 2023-11-01T04:16:09.873 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-11-01T04:16:09.883 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:09.883 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:09.883 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:16:09.883 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:16:09.884 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:09.892 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2023-11-01T04:16:09.896 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:09.890+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:09.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-01T04:16:09.984 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:09.984 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-01T04:16:09.997 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:09.994+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:10.006 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:10.006 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-01T04:16:10.024 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:10.024 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-01T04:16:10.063 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:10.063 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:16:10.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:16:10.063 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [] 2023-11-01T04:16:10.063 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:10.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-01T04:16:10.098 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:10.096+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:10.199 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:16:10.200+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:16:10.242 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:10.242 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:16:10.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.243 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.4241.0:188 1.0 e31/23)", 2023-11-01T04:16:10.243 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:10.070091+0000", 2023-11-01T04:16:10.243 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.17317064200000001, 2023-11-01T04:16:10.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17320788400000001, 2023-11-01T04:16:10.243 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.244 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:10.244 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:10.070091+0000", 2023-11-01T04:16:10.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:10.070091+0000", 2023-11-01T04:16:10.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:10.070092+0000", 2023-11-01T04:16:10.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5300000000000003e-07 2023-11-01T04:16:10.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:10.070098+0000", 2023-11-01T04:16:10.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7270000000000004e-06 2023-11-01T04:16:10.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:10.070116+0000", 2023-11-01T04:16:10.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8779000000000001e-05 2023-11-01T04:16:10.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:10.070126+0000", 2023-11-01T04:16:10.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.876e-06 2023-11-01T04:16:10.250 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.250 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.250 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.251 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.4241.0:189 1.0 e31/23)", 2023-11-01T04:16:10.251 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:10.166030+0000", 2023-11-01T04:16:10.251 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.077231937000000001, 2023-11-01T04:16:10.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.077351907999999997, 2023-11-01T04:16:10.251 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.252 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:10.252 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:10.166030+0000", 2023-11-01T04:16:10.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:10.166030+0000", 2023-11-01T04:16:10.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:10.166032+0000", 2023-11-01T04:16:10.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9929999999999998e-06 2023-11-01T04:16:10.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:10.166044+0000", 2023-11-01T04:16:10.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2099000000000001e-05 2023-11-01T04:16:10.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:10.166075+0000", 2023-11-01T04:16:10.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1241999999999997e-05 2023-11-01T04:16:10.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:10.166098+0000", 2023-11-01T04:16:10.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2693000000000001e-05 2023-11-01T04:16:10.257 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.258 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.258 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.258 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.258 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2023-11-01T04:16:10.258 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 2 2023-11-01T04:16:10.258 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:10.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-01T04:16:10.300 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:10.300+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:10.381 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:10.381 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:10.381 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:16:10.381 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:16:10.382 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:10.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-01T04:16:10.401 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:10.400+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:10.501 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:10.498+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:10.521 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:16:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:16:10.521 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:16:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2023-11-01T04:16:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:03.486494+0000", 2023-11-01T04:16:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 7.0313644149999996, 2023-11-01T04:16:10.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50355865899999996, 2023-11-01T04:16:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.523 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.524 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87 2023-11-01T04:16:10.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.524 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.525 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:03.486494+0000", 2023-11-01T04:16:10.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:03.486494+0000", 2023-11-01T04:16:10.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:03.486495+0000", 2023-11-01T04:16:10.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3829999999999999e-06 2023-11-01T04:16:10.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:03.486644+0000", 2023-11-01T04:16:10.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014873800000000001 2023-11-01T04:16:10.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:03.486668+0000", 2023-11-01T04:16:10.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3538999999999999e-05 2023-11-01T04:16:10.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:03.586773+0000", 2023-11-01T04:16:10.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10010553899999999 2023-11-01T04:16:10.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:03.586922+0000", 2023-11-01T04:16:10.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014889800000000001 2023-11-01T04:16:10.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:03.587184+0000", 2023-11-01T04:16:10.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026156500000000001 2023-11-01T04:16:10.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:03.587559+0000", 2023-11-01T04:16:10.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00037534599999999998 2023-11-01T04:16:10.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:03.589969+0000", 2023-11-01T04:16:10.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024101690000000002 2023-11-01T04:16:10.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:03.989903+0000", 2023-11-01T04:16:10.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39993345299999999 2023-11-01T04:16:10.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:03.989989+0000", 2023-11-01T04:16:10.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6751999999999999e-05 2023-11-01T04:16:10.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:03.990053+0000", 2023-11-01T04:16:10.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3276999999999997e-05 2023-11-01T04:16:10.537 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.537 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.537 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.538 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.538 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138478+0000", 2023-11-01T04:16:10.538 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.3793804310000004, 2023-11-01T04:16:10.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51616329400000005, 2023-11-01T04:16:10.538 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.539 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.539 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.539 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.539 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.539 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 111 2023-11-01T04:16:10.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.540 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138478+0000", 2023-11-01T04:16:10.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138478+0000", 2023-11-01T04:16:10.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138479+0000", 2023-11-01T04:16:10.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6000000000000003e-07 2023-11-01T04:16:10.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138484+0000", 2023-11-01T04:16:10.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5099999999999998e-06 2023-11-01T04:16:10.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138494+0000", 2023-11-01T04:16:10.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3659999999999998e-06 2023-11-01T04:16:10.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138501+0000", 2023-11-01T04:16:10.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7230000000000004e-06 2023-11-01T04:16:10.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.243922+0000", 2023-11-01T04:16:10.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105420454 2023-11-01T04:16:10.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.244148+0000", 2023-11-01T04:16:10.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022663800000000001 2023-11-01T04:16:10.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.244505+0000", 2023-11-01T04:16:10.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00035644499999999999 2023-11-01T04:16:10.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.250339+0000", 2023-11-01T04:16:10.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058344959999999998 2023-11-01T04:16:10.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.654434+0000", 2023-11-01T04:16:10.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40409476900000002 2023-11-01T04:16:10.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.654518+0000", 2023-11-01T04:16:10.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4421999999999999e-05 2023-11-01T04:16:10.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.654641+0000", 2023-11-01T04:16:10.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012281099999999999 2023-11-01T04:16:10.552 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.552 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.552 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.553 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.553 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138513+0000", 2023-11-01T04:16:10.553 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.3793458430000003, 2023-11-01T04:16:10.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51640948099999995, 2023-11-01T04:16:10.553 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.553 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.554 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.554 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.554 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.554 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 112 2023-11-01T04:16:10.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.555 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138513+0000", 2023-11-01T04:16:10.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138513+0000", 2023-11-01T04:16:10.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138513+0000", 2023-11-01T04:16:10.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5000000000000002e-07 2023-11-01T04:16:10.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138680+0000", 2023-11-01T04:16:10.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016725400000000001 2023-11-01T04:16:10.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138694+0000", 2023-11-01T04:16:10.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3495999999999999e-05 2023-11-01T04:16:10.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138706+0000", 2023-11-01T04:16:10.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1557e-05 2023-11-01T04:16:10.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.245369+0000", 2023-11-01T04:16:10.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10666384600000001 2023-11-01T04:16:10.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.245509+0000", 2023-11-01T04:16:10.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014007900000000001 2023-11-01T04:16:10.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.245728+0000", 2023-11-01T04:16:10.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000218345 2023-11-01T04:16:10.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352473+0000", 2023-11-01T04:16:10.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.106745693 2023-11-01T04:16:10.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.654776+0000", 2023-11-01T04:16:10.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30230230200000002 2023-11-01T04:16:10.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.654863+0000", 2023-11-01T04:16:10.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7638000000000004e-05 2023-11-01T04:16:10.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.654922+0000", 2023-11-01T04:16:10.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8621e-05 2023-11-01T04:16:10.567 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.567 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.567 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.568 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.568 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138723+0000", 2023-11-01T04:16:10.568 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.3791357470000003, 2023-11-01T04:16:10.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51647989999999999, 2023-11-01T04:16:10.569 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.569 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.569 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.569 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.569 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.570 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 113 2023-11-01T04:16:10.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.570 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138723+0000", 2023-11-01T04:16:10.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138723+0000", 2023-11-01T04:16:10.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138724+0000", 2023-11-01T04:16:10.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5600000000000004e-07 2023-11-01T04:16:10.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138729+0000", 2023-11-01T04:16:10.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7799999999999997e-06 2023-11-01T04:16:10.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138741+0000", 2023-11-01T04:16:10.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1846e-05 2023-11-01T04:16:10.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138749+0000", 2023-11-01T04:16:10.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0499999999999992e-06 2023-11-01T04:16:10.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.246597+0000", 2023-11-01T04:16:10.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10784767300000001 2023-11-01T04:16:10.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.246820+0000", 2023-11-01T04:16:10.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022339199999999999 2023-11-01T04:16:10.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.247168+0000", 2023-11-01T04:16:10.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000347521 2023-11-01T04:16:10.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352480+0000", 2023-11-01T04:16:10.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105312482 2023-11-01T04:16:10.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.655031+0000", 2023-11-01T04:16:10.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30255108200000003 2023-11-01T04:16:10.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.655110+0000", 2023-11-01T04:16:10.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8678999999999997e-05 2023-11-01T04:16:10.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.655203+0000", 2023-11-01T04:16:10.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2439e-05 2023-11-01T04:16:10.582 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.582 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.582 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.583 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.583 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138761+0000", 2023-11-01T04:16:10.583 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.3790971760000001, 2023-11-01T04:16:10.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61681865999999996, 2023-11-01T04:16:10.584 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.584 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.584 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.584 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.584 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.584 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 114 2023-11-01T04:16:10.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.585 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138761+0000", 2023-11-01T04:16:10.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138761+0000", 2023-11-01T04:16:10.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138762+0000", 2023-11-01T04:16:10.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4300000000000003e-07 2023-11-01T04:16:10.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138768+0000", 2023-11-01T04:16:10.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7729999999999996e-06 2023-11-01T04:16:10.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138777+0000", 2023-11-01T04:16:10.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1929999999999999e-06 2023-11-01T04:16:10.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138784+0000", 2023-11-01T04:16:10.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3830000000000003e-06 2023-11-01T04:16:10.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.248094+0000", 2023-11-01T04:16:10.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10930951699999999 2023-11-01T04:16:10.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.248210+0000", 2023-11-01T04:16:10.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011579299999999999 2023-11-01T04:16:10.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.248407+0000", 2023-11-01T04:16:10.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019723000000000001 2023-11-01T04:16:10.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352490+0000", 2023-11-01T04:16:10.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104083393 2023-11-01T04:16:10.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755390+0000", 2023-11-01T04:16:10.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40290009500000001 2023-11-01T04:16:10.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755474+0000", 2023-11-01T04:16:10.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3323000000000002e-05 2023-11-01T04:16:10.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755580+0000", 2023-11-01T04:16:10.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010631700000000001 2023-11-01T04:16:10.597 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.597 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.597 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.597 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.598 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138795+0000", 2023-11-01T04:16:10.598 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.3790630180000001, 2023-11-01T04:16:10.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61707305000000001, 2023-11-01T04:16:10.598 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.598 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.599 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.599 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.599 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.599 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 115 2023-11-01T04:16:10.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.599 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138795+0000", 2023-11-01T04:16:10.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138795+0000", 2023-11-01T04:16:10.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138796+0000", 2023-11-01T04:16:10.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3300000000000002e-07 2023-11-01T04:16:10.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138931+0000", 2023-11-01T04:16:10.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013525000000000001 2023-11-01T04:16:10.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138947+0000", 2023-11-01T04:16:10.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5478999999999999e-05 2023-11-01T04:16:10.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138956+0000", 2023-11-01T04:16:10.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1530000000000003e-06 2023-11-01T04:16:10.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.249005+0000", 2023-11-01T04:16:10.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.110048914 2023-11-01T04:16:10.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.249150+0000", 2023-11-01T04:16:10.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014491600000000001 2023-11-01T04:16:10.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.249351+0000", 2023-11-01T04:16:10.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000201516 2023-11-01T04:16:10.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352738+0000", 2023-11-01T04:16:10.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10338632 2023-11-01T04:16:10.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755668+0000", 2023-11-01T04:16:10.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40293035700000002 2023-11-01T04:16:10.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755765+0000", 2023-11-01T04:16:10.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7081000000000001e-05 2023-11-01T04:16:10.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755868+0000", 2023-11-01T04:16:10.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010343100000000001 2023-11-01T04:16:10.613 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.613 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.613 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.614 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.614 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138971+0000", 2023-11-01T04:16:10.614 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.3788874800000004, 2023-11-01T04:16:10.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71733164100000002, 2023-11-01T04:16:10.615 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.615 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.615 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.616 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.616 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.616 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 116 2023-11-01T04:16:10.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.616 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138971+0000", 2023-11-01T04:16:10.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138971+0000", 2023-11-01T04:16:10.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138972+0000", 2023-11-01T04:16:10.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4000000000000001e-07 2023-11-01T04:16:10.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138988+0000", 2023-11-01T04:16:10.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6354999999999998e-05 2023-11-01T04:16:10.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138998+0000", 2023-11-01T04:16:10.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0285999999999999e-05 2023-11-01T04:16:10.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139010+0000", 2023-11-01T04:16:10.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1807000000000001e-05 2023-11-01T04:16:10.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.350520+0000", 2023-11-01T04:16:10.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21151004200000001 2023-11-01T04:16:10.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.350668+0000", 2023-11-01T04:16:10.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014824900000000001 2023-11-01T04:16:10.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.350873+0000", 2023-11-01T04:16:10.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020507899999999999 2023-11-01T04:16:10.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352744+0000", 2023-11-01T04:16:10.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001870741 2023-11-01T04:16:10.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856134+0000", 2023-11-01T04:16:10.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50339015399999998 2023-11-01T04:16:10.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856235+0000", 2023-11-01T04:16:10.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010087200000000001 2023-11-01T04:16:10.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856303+0000", 2023-11-01T04:16:10.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7316000000000004e-05 2023-11-01T04:16:10.629 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.629 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.629 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.630 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.630 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.139035+0000", 2023-11-01T04:16:10.630 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.3788234939999997, 2023-11-01T04:16:10.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71770837499999995, 2023-11-01T04:16:10.631 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.631 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.631 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.631 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.632 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.632 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 117 2023-11-01T04:16:10.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.632 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139035+0000", 2023-11-01T04:16:10.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139035+0000", 2023-11-01T04:16:10.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139036+0000", 2023-11-01T04:16:10.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7999999999999995e-07 2023-11-01T04:16:10.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139042+0000", 2023-11-01T04:16:10.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9499999999999998e-06 2023-11-01T04:16:10.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139051+0000", 2023-11-01T04:16:10.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7489999999999993e-06 2023-11-01T04:16:10.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139059+0000", 2023-11-01T04:16:10.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8159999999999994e-06 2023-11-01T04:16:10.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.351436+0000", 2023-11-01T04:16:10.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21237641400000001 2023-11-01T04:16:10.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.351583+0000", 2023-11-01T04:16:10.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000147186 2023-11-01T04:16:10.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.351772+0000", 2023-11-01T04:16:10.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018879300000000001 2023-11-01T04:16:10.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.553963+0000", 2023-11-01T04:16:10.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20219130599999999 2023-11-01T04:16:10.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856458+0000", 2023-11-01T04:16:10.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30249546300000002 2023-11-01T04:16:10.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856632+0000", 2023-11-01T04:16:10.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000173611 2023-11-01T04:16:10.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856743+0000", 2023-11-01T04:16:10.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111407 2023-11-01T04:16:10.644 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.644 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.644 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.645 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:130 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.645 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.621326+0000", 2023-11-01T04:16:10.645 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.8965325330000002, 2023-11-01T04:16:10.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50392313600000005, 2023-11-01T04:16:10.646 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.646 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.646 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.646 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.646 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.647 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 130 2023-11-01T04:16:10.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.647 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621326+0000", 2023-11-01T04:16:10.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621326+0000", 2023-11-01T04:16:10.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621327+0000", 2023-11-01T04:16:10.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5200000000000001e-06 2023-11-01T04:16:10.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621342+0000", 2023-11-01T04:16:10.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4449e-05 2023-11-01T04:16:10.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621355+0000", 2023-11-01T04:16:10.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.291e-05 2023-11-01T04:16:10.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621365+0000", 2023-11-01T04:16:10.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9890000000000004e-06 2023-11-01T04:16:10.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621962+0000", 2023-11-01T04:16:10.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00059754299999999997 2023-11-01T04:16:10.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622048+0000", 2023-11-01T04:16:10.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5742999999999995e-05 2023-11-01T04:16:10.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622177+0000", 2023-11-01T04:16:10.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012937100000000001 2023-11-01T04:16:10.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835732+0000", 2023-11-01T04:16:10.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21355484799999999 2023-11-01T04:16:10.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.125109+0000", 2023-11-01T04:16:10.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28937690799999999 2023-11-01T04:16:10.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.125192+0000", 2023-11-01T04:16:10.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2421000000000003e-05 2023-11-01T04:16:10.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.125249+0000", 2023-11-01T04:16:10.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7433999999999999e-05 2023-11-01T04:16:10.660 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.660 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.661 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.661 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:131 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.661 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.621413+0000", 2023-11-01T04:16:10.661 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.8964450479999999, 2023-11-01T04:16:10.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50410262299999997, 2023-11-01T04:16:10.662 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.662 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.662 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.662 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.663 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.663 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 131 2023-11-01T04:16:10.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.663 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621413+0000", 2023-11-01T04:16:10.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621413+0000", 2023-11-01T04:16:10.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621415+0000", 2023-11-01T04:16:10.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2699999999999999e-06 2023-11-01T04:16:10.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621583+0000", 2023-11-01T04:16:10.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016864500000000001 2023-11-01T04:16:10.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621598+0000", 2023-11-01T04:16:10.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4262e-05 2023-11-01T04:16:10.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621607+0000", 2023-11-01T04:16:10.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9029999999999999e-06 2023-11-01T04:16:10.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622526+0000", 2023-11-01T04:16:10.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091876800000000004 2023-11-01T04:16:10.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622628+0000", 2023-11-01T04:16:10.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101413 2023-11-01T04:16:10.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622761+0000", 2023-11-01T04:16:10.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000133781 2023-11-01T04:16:10.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835742+0000", 2023-11-01T04:16:10.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21298055900000001 2023-11-01T04:16:10.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.125351+0000", 2023-11-01T04:16:10.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28960872700000001 2023-11-01T04:16:10.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.125448+0000", 2023-11-01T04:16:10.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6947999999999994e-05 2023-11-01T04:16:10.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.125516+0000", 2023-11-01T04:16:10.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8347000000000001e-05 2023-11-01T04:16:10.675 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.675 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.675 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.676 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.676 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.621630+0000", 2023-11-01T04:16:10.676 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.8962281459999999, 2023-11-01T04:16:10.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50409248500000003, 2023-11-01T04:16:10.677 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.677 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.677 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.677 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.677 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.677 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 132 2023-11-01T04:16:10.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.678 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621630+0000", 2023-11-01T04:16:10.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621630+0000", 2023-11-01T04:16:10.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621631+0000", 2023-11-01T04:16:10.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0699999999999996e-07 2023-11-01T04:16:10.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621637+0000", 2023-11-01T04:16:10.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7599999999999999e-06 2023-11-01T04:16:10.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621649+0000", 2023-11-01T04:16:10.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1755e-05 2023-11-01T04:16:10.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621657+0000", 2023-11-01T04:16:10.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9540000000000004e-06 2023-11-01T04:16:10.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.623191+0000", 2023-11-01T04:16:10.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001534828 2023-11-01T04:16:10.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.623279+0000", 2023-11-01T04:16:10.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7886999999999994e-05 2023-11-01T04:16:10.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.623416+0000", 2023-11-01T04:16:10.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000136976 2023-11-01T04:16:10.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835749+0000", 2023-11-01T04:16:10.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21233226799999999 2023-11-01T04:16:10.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.125602+0000", 2023-11-01T04:16:10.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.289853156 2023-11-01T04:16:10.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.125675+0000", 2023-11-01T04:16:10.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3242000000000001e-05 2023-11-01T04:16:10.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.125723+0000", 2023-11-01T04:16:10.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7752e-05 2023-11-01T04:16:10.690 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.690 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.690 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.691 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.691 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.621912+0000", 2023-11-01T04:16:10.691 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.8959466840000001, 2023-11-01T04:16:10.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50401969800000002, 2023-11-01T04:16:10.691 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.692 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.692 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.692 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.692 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.692 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 133 2023-11-01T04:16:10.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.693 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621912+0000", 2023-11-01T04:16:10.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621912+0000", 2023-11-01T04:16:10.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621913+0000", 2023-11-01T04:16:10.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.44e-06 2023-11-01T04:16:10.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621921+0000", 2023-11-01T04:16:10.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9130000000000003e-06 2023-11-01T04:16:10.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621936+0000", 2023-11-01T04:16:10.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4812e-05 2023-11-01T04:16:10.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621945+0000", 2023-11-01T04:16:10.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4870000000000008e-06 2023-11-01T04:16:10.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.623759+0000", 2023-11-01T04:16:10.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001813914 2023-11-01T04:16:10.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.623840+0000", 2023-11-01T04:16:10.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0785000000000007e-05 2023-11-01T04:16:10.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.624007+0000", 2023-11-01T04:16:10.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016651 2023-11-01T04:16:10.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835757+0000", 2023-11-01T04:16:10.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21175090199999999 2023-11-01T04:16:10.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.125790+0000", 2023-11-01T04:16:10.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29003205199999998 2023-11-01T04:16:10.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.125876+0000", 2023-11-01T04:16:10.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6621999999999998e-05 2023-11-01T04:16:10.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.125931+0000", 2023-11-01T04:16:10.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5260999999999998e-05 2023-11-01T04:16:10.705 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.705 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.705 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.706 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.706 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622060+0000", 2023-11-01T04:16:10.706 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.895798825, 2023-11-01T04:16:10.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.504069447, 2023-11-01T04:16:10.706 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.707 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.707 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.707 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.707 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.707 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 134 2023-11-01T04:16:10.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.708 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622060+0000", 2023-11-01T04:16:10.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622060+0000", 2023-11-01T04:16:10.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622060+0000", 2023-11-01T04:16:10.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7000000000000003e-07 2023-11-01T04:16:10.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622221+0000", 2023-11-01T04:16:10.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016092799999999999 2023-11-01T04:16:10.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622250+0000", 2023-11-01T04:16:10.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8544999999999999e-05 2023-11-01T04:16:10.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622260+0000", 2023-11-01T04:16:10.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9000000000000001e-06 2023-11-01T04:16:10.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.624364+0000", 2023-11-01T04:16:10.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021042750000000001 2023-11-01T04:16:10.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.624447+0000", 2023-11-01T04:16:10.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2882999999999999e-05 2023-11-01T04:16:10.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.624585+0000", 2023-11-01T04:16:10.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000137529 2023-11-01T04:16:10.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835767+0000", 2023-11-01T04:16:10.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21118271299999999 2023-11-01T04:16:10.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126007+0000", 2023-11-01T04:16:10.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29023952400000003 2023-11-01T04:16:10.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126080+0000", 2023-11-01T04:16:10.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3555999999999998e-05 2023-11-01T04:16:10.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126129+0000", 2023-11-01T04:16:10.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8724000000000003e-05 2023-11-01T04:16:10.720 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.720 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.720 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.721 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.721 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622286+0000", 2023-11-01T04:16:10.721 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.8955723600000001, 2023-11-01T04:16:10.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50406131600000004, 2023-11-01T04:16:10.722 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.722 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.722 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.722 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.722 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.723 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 135 2023-11-01T04:16:10.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.723 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622286+0000", 2023-11-01T04:16:10.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622286+0000", 2023-11-01T04:16:10.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622287+0000", 2023-11-01T04:16:10.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7599999999999996e-07 2023-11-01T04:16:10.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622293+0000", 2023-11-01T04:16:10.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8429999999999998e-06 2023-11-01T04:16:10.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622304+0000", 2023-11-01T04:16:10.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1147e-05 2023-11-01T04:16:10.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622312+0000", 2023-11-01T04:16:10.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9920000000000007e-06 2023-11-01T04:16:10.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.729 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.625031+0000", 2023-11-01T04:16:10.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027192190000000001 2023-11-01T04:16:10.730 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.730 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.730 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.625121+0000", 2023-11-01T04:16:10.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9835999999999998e-05 2023-11-01T04:16:10.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.731 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.625260+0000", 2023-11-01T04:16:10.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013933399999999999 2023-11-01T04:16:10.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835776+0000", 2023-11-01T04:16:10.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21051568800000001 2023-11-01T04:16:10.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126207+0000", 2023-11-01T04:16:10.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29043097099999998 2023-11-01T04:16:10.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.734 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126291+0000", 2023-11-01T04:16:10.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4133000000000002e-05 2023-11-01T04:16:10.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.735 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126347+0000", 2023-11-01T04:16:10.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6276999999999997e-05 2023-11-01T04:16:10.735 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.735 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.735 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.736 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.736 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622325+0000", 2023-11-01T04:16:10.736 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.895533806, 2023-11-01T04:16:10.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50422977499999999, 2023-11-01T04:16:10.737 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.737 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.737 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.737 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.738 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.738 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 136 2023-11-01T04:16:10.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.738 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622325+0000", 2023-11-01T04:16:10.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622325+0000", 2023-11-01T04:16:10.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622325+0000", 2023-11-01T04:16:10.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7400000000000003e-07 2023-11-01T04:16:10.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622330+0000", 2023-11-01T04:16:10.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8720000000000001e-06 2023-11-01T04:16:10.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622340+0000", 2023-11-01T04:16:10.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6739999999999992e-06 2023-11-01T04:16:10.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622347+0000", 2023-11-01T04:16:10.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3660000000000003e-06 2023-11-01T04:16:10.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.744 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.725754+0000", 2023-11-01T04:16:10.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.103406496 2023-11-01T04:16:10.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.726001+0000", 2023-11-01T04:16:10.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024736099999999998 2023-11-01T04:16:10.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.726371+0000", 2023-11-01T04:16:10.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00037005099999999998 2023-11-01T04:16:10.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835786+0000", 2023-11-01T04:16:10.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.109414463 2023-11-01T04:16:10.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126424+0000", 2023-11-01T04:16:10.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29063866999999999 2023-11-01T04:16:10.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.749 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126500+0000", 2023-11-01T04:16:10.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6212999999999997e-05 2023-11-01T04:16:10.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126554+0000", 2023-11-01T04:16:10.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3934999999999998e-05 2023-11-01T04:16:10.750 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.750 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.750 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.751 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.751 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622359+0000", 2023-11-01T04:16:10.751 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.8954998710000002, 2023-11-01T04:16:10.752 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50439429300000005, 2023-11-01T04:16:10.752 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.752 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.752 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.752 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.752 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.753 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 137 2023-11-01T04:16:10.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.753 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.753 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622359+0000", 2023-11-01T04:16:10.754 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.754 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622359+0000", 2023-11-01T04:16:10.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.756 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622359+0000", 2023-11-01T04:16:10.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.44e-07 2023-11-01T04:16:10.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.756 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.756 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622516+0000", 2023-11-01T04:16:10.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000156945 2023-11-01T04:16:10.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.757 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622547+0000", 2023-11-01T04:16:10.758 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0747999999999998e-05 2023-11-01T04:16:10.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.758 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.758 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.758 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622564+0000", 2023-11-01T04:16:10.759 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7516e-05 2023-11-01T04:16:10.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.759 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.759 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.759 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.727341+0000", 2023-11-01T04:16:10.759 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104776878 2023-11-01T04:16:10.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.760 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.760 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.760 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.727568+0000", 2023-11-01T04:16:10.760 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000226908 2023-11-01T04:16:10.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.761 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.761 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.761 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.727955+0000", 2023-11-01T04:16:10.761 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038721900000000001 2023-11-01T04:16:10.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.762 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.762 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.762 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835801+0000", 2023-11-01T04:16:10.762 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.107845343 2023-11-01T04:16:10.762 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.762 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.763 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.763 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126619+0000", 2023-11-01T04:16:10.763 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.290817879 2023-11-01T04:16:10.763 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.763 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.764 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.764 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126696+0000", 2023-11-01T04:16:10.764 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7571999999999995e-05 2023-11-01T04:16:10.764 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.764 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.764 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.765 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126753+0000", 2023-11-01T04:16:10.765 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6641000000000002e-05 2023-11-01T04:16:10.765 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.765 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.765 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.765 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.766 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.766 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.766 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622584+0000", 2023-11-01T04:16:10.766 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.8952740430000001, 2023-11-01T04:16:10.766 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50666713100000005, 2023-11-01T04:16:10.767 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.767 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.767 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.767 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.767 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.767 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 138 2023-11-01T04:16:10.768 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.768 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.768 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.768 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.768 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622584+0000", 2023-11-01T04:16:10.769 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.769 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.769 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.769 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622584+0000", 2023-11-01T04:16:10.770 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.770 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.770 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.770 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622585+0000", 2023-11-01T04:16:10.770 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0640000000000001e-06 2023-11-01T04:16:10.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.771 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.771 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.771 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622591+0000", 2023-11-01T04:16:10.771 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9059999999999999e-06 2023-11-01T04:16:10.772 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.772 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.772 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.772 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622602+0000", 2023-11-01T04:16:10.772 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0309000000000001e-05 2023-11-01T04:16:10.772 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.773 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.773 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.773 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622609+0000", 2023-11-01T04:16:10.773 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4869999999999996e-06 2023-11-01T04:16:10.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.773 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.774 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.774 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.729780+0000", 2023-11-01T04:16:10.774 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10717114799999999 2023-11-01T04:16:10.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.774 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.775 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.775 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.730026+0000", 2023-11-01T04:16:10.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024591700000000001 2023-11-01T04:16:10.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.775 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.775 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.776 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.730386+0000", 2023-11-01T04:16:10.776 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036019099999999999 2023-11-01T04:16:10.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.776 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.776 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.776 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835810+0000", 2023-11-01T04:16:10.777 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105423431 2023-11-01T04:16:10.777 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.777 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.777 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.777 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.129130+0000", 2023-11-01T04:16:10.777 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29331975399999999 2023-11-01T04:16:10.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.778 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.778 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.778 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.129203+0000", 2023-11-01T04:16:10.778 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3418999999999996e-05 2023-11-01T04:16:10.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.779 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.779 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.779 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.129252+0000", 2023-11-01T04:16:10.779 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8504999999999997e-05 2023-11-01T04:16:10.779 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.780 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.780 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.780 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.780 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.780 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622627+0000", 2023-11-01T04:16:10.781 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.8952317949999999, 2023-11-01T04:16:10.781 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60704219400000003, 2023-11-01T04:16:10.781 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.781 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.781 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.782 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.782 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.782 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 139 2023-11-01T04:16:10.782 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.782 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.782 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.783 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.783 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622627+0000", 2023-11-01T04:16:10.783 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.783 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.783 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.783 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.784 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622627+0000", 2023-11-01T04:16:10.784 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.784 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.784 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.784 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.785 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622627+0000", 2023-11-01T04:16:10.785 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5400000000000001e-07 2023-11-01T04:16:10.785 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.785 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.785 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.785 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622632+0000", 2023-11-01T04:16:10.786 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0420000000000002e-06 2023-11-01T04:16:10.786 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.786 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.786 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.786 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622642+0000", 2023-11-01T04:16:10.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.713e-06 2023-11-01T04:16:10.787 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.787 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.787 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.787 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622649+0000", 2023-11-01T04:16:10.788 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4460000000000004e-06 2023-11-01T04:16:10.788 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.788 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.831442+0000", 2023-11-01T04:16:10.788 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20879259 2023-11-01T04:16:10.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.789 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.789 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.831689+0000", 2023-11-01T04:16:10.789 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000247087 2023-11-01T04:16:10.790 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.790 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.790 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.790 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.832051+0000", 2023-11-01T04:16:10.790 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036175699999999999 2023-11-01T04:16:10.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.791 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.791 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.791 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.837325+0000", 2023-11-01T04:16:10.791 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0052741289999999998 2023-11-01T04:16:10.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.792 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229409+0000", 2023-11-01T04:16:10.792 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39208416000000001 2023-11-01T04:16:10.792 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.793 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.793 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.793 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229566+0000", 2023-11-01T04:16:10.793 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015727499999999999 2023-11-01T04:16:10.793 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.793 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.794 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.794 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229669+0000", 2023-11-01T04:16:10.794 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000102341 2023-11-01T04:16:10.794 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.794 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.794 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.795 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.795 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.795 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.795 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622661+0000", 2023-11-01T04:16:10.795 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.8951974030000001, 2023-11-01T04:16:10.796 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60735416200000003, 2023-11-01T04:16:10.796 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.796 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.796 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.796 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.796 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.797 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 140 2023-11-01T04:16:10.797 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.797 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.797 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.797 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.798 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622661+0000", 2023-11-01T04:16:10.798 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.798 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.798 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.798 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.798 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622661+0000", 2023-11-01T04:16:10.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.799 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.799 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.799 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.799 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622662+0000", 2023-11-01T04:16:10.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3e-07 2023-11-01T04:16:10.800 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.800 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.800 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.800 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622803+0000", 2023-11-01T04:16:10.800 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014142199999999999 2023-11-01T04:16:10.800 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.801 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.801 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.801 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622816+0000", 2023-11-01T04:16:10.801 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2683e-05 2023-11-01T04:16:10.801 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.802 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.802 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622825+0000", 2023-11-01T04:16:10.802 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9959999999999999e-06 2023-11-01T04:16:10.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.833052+0000", 2023-11-01T04:16:10.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21022711699999999 2023-11-01T04:16:10.803 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.804 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.833201+0000", 2023-11-01T04:16:10.804 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000149059 2023-11-01T04:16:10.804 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.804 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.804 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.833405+0000", 2023-11-01T04:16:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020363600000000001 2023-11-01T04:16:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.805 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.837343+0000", 2023-11-01T04:16:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039379590000000004 2023-11-01T04:16:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229871+0000", 2023-11-01T04:16:10.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39252865399999998 2023-11-01T04:16:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229957+0000", 2023-11-01T04:16:10.807 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5962000000000001e-05 2023-11-01T04:16:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.230015+0000", 2023-11-01T04:16:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8044000000000002e-05 2023-11-01T04:16:10.808 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:10.809 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622843+0000", 2023-11-01T04:16:10.810 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.895015077, 2023-11-01T04:16:10.810 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70733220600000002, 2023-11-01T04:16:10.810 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:10.810 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:10.810 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 141 2023-11-01T04:16:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:10.811 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622843+0000", 2023-11-01T04:16:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.812 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622843+0000", 2023-11-01T04:16:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622844+0000", 2023-11-01T04:16:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7400000000000002e-07 2023-11-01T04:16:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:10.814 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622850+0000", 2023-11-01T04:16:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4330000000000003e-06 2023-11-01T04:16:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622859+0000", 2023-11-01T04:16:10.815 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5829999999999996e-06 2023-11-01T04:16:10.816 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.816 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.816 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:10.816 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.722958+0000", 2023-11-01T04:16:10.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100098824 2023-11-01T04:16:10.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:10.817 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835872+0000", 2023-11-01T04:16:10.817 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.112913586 2023-11-01T04:16:10.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.818 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.818 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:10.818 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.836107+0000", 2023-11-01T04:16:10.818 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023507399999999999 2023-11-01T04:16:10.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:10.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.836474+0000", 2023-11-01T04:16:10.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036769800000000002 2023-11-01T04:16:10.819 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.820 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:10.820 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.837993+0000", 2023-11-01T04:16:10.820 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001518192 2023-11-01T04:16:10.820 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.820 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:10.821 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.329913+0000", 2023-11-01T04:16:10.821 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.49192055499999998 2023-11-01T04:16:10.821 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.821 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.821 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:10.822 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.330067+0000", 2023-11-01T04:16:10.822 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000154012 2023-11-01T04:16:10.822 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:10.822 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:10.822 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:10.822 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.330176+0000", 2023-11-01T04:16:10.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010847499999999999 2023-11-01T04:16:10.823 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.823 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.823 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.823 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:10.823 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:10.824 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:10.825 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-01T04:16:10.826 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:10.824+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:10.887 INFO:teuthology.orchestra.run.smithi153.stderr:enabled application 'rados' on pool 'unique_pool_0' 2023-11-01T04:16:10.887 INFO:teuthology.orchestra.run.smithi153.stderr:nodeep-scrub is set 2023-11-01T04:16:10.959 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:10.959 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:16:10.959 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2023-11-01T04:16:10.960 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:10.963 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:10.961+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:10.973 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-01T04:16:11.026 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:11.024+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:11.093 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:11.093 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:16:11.093 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2023-11-01T04:16:11.093 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2023-11-01T04:16:11.093 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:11.108 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-01T04:16:11.128 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:16:11.128+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:16:11.229 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:11.225+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:11.276 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:11.277 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2023-11-01T04:16:11.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2023-11-01T04:16:11.277 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [] 2023-11-01T04:16:11.277 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:11.293 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-01T04:16:11.330 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:11.328+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:11.413 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:11.413 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:16:11.413 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2023-11-01T04:16:11.413 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:11.427 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-01T04:16:11.430 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:11.425+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:11.530 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:11.528+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:11.545 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:11.545 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:16:11.545 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2023-11-01T04:16:11.546 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2023-11-01T04:16:11.546 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:11.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-01T04:16:11.631 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:11.632+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:11.675 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2023-11-01T04:16:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2023-11-01T04:16:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [] 2023-11-01T04:16:11.676 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:11.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-01T04:16:11.732 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:11.729+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:11.811 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:11.811 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:16:11.811 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2023-11-01T04:16:11.812 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:11.826 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-01T04:16:11.833 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:11.829+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:11.922 DEBUG:teuthology.orchestra.run.smithi190:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2023-11-01T04:16:11.934 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:11.934+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:11.941 INFO:tasks.rados.rados.0.smithi190.stdout:adding op weight read -> 100 2023-11-01T04:16:11.942 INFO:tasks.rados.rados.0.smithi190.stdout:adding op weight delete -> 50 2023-11-01T04:16:11.942 INFO:tasks.rados.rados.0.smithi190.stdout:adding op weight snap_create -> 50 2023-11-01T04:16:11.942 INFO:tasks.rados.rados.0.smithi190.stdout:adding op weight snap_remove -> 50 2023-11-01T04:16:11.942 INFO:tasks.rados.rados.0.smithi190.stdout:adding op weight rollback -> 50 2023-11-01T04:16:11.942 INFO:tasks.rados.rados.0.smithi190.stdout:adding op weight setattr -> 25 2023-11-01T04:16:11.943 INFO:tasks.rados.rados.0.smithi190.stdout:adding op weight rmattr -> 25 2023-11-01T04:16:11.943 INFO:tasks.rados.rados.0.smithi190.stdout:adding op weight copy_from -> 50 2023-11-01T04:16:11.943 INFO:tasks.rados.rados.0.smithi190.stdout:adding op weight append -> 100 2023-11-01T04:16:11.943 INFO:tasks.rados.rados.0.smithi190.stdout:ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:11.943 INFO:tasks.rados.rados.0.smithi190.stdout:Configuration: 2023-11-01T04:16:11.944 INFO:tasks.rados.rados.0.smithi190.stdout: Number of operations: 400 2023-11-01T04:16:11.944 INFO:tasks.rados.rados.0.smithi190.stdout: Number of objects: 50 2023-11-01T04:16:11.944 INFO:tasks.rados.rados.0.smithi190.stdout: Max in flight operations: 16 2023-11-01T04:16:11.944 INFO:tasks.rados.rados.0.smithi190.stdout: Object size (in bytes): 4000000 2023-11-01T04:16:11.945 INFO:tasks.rados.rados.0.smithi190.stdout: Write stride min: 400000 2023-11-01T04:16:11.945 INFO:tasks.rados.rados.0.smithi190.stdout: Write stride max: 800000 2023-11-01T04:16:11.953 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:11.953 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:16:11.953 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2023-11-01T04:16:11.954 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2023-11-01T04:16:11.954 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:11.968 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-01T04:16:12.036 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:12.030+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:12.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-01T04:16:12.090 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:12.090 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2023-11-01T04:16:12.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2023-11-01T04:16:12.091 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [] 2023-11-01T04:16:12.091 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:12.104 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-01T04:16:12.117 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2023-11-01T04:16:12.117 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-01T04:16:12.118 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-11-01T04:16:12.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-11-01T04:16:12.136 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:12.136+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:12.200 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:12.200 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-11-01T04:16:12.220 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:12.220 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-01T04:16:12.227 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:12.228 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:16:12.228 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2023-11-01T04:16:12.228 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:12.237 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:12.233+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:12.238 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:12.238 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-01T04:16:12.243 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-01T04:16:12.260 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:12.260 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-01T04:16:12.276 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:12.276 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-01T04:16:12.309 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:12.310 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-01T04:16:12.338 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:12.333+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:12.341 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:12.341 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-01T04:16:12.358 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:12.358 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:16:12.358 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2023-11-01T04:16:12.359 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2023-11-01T04:16:12.359 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:12.361 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:12.362 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-01T04:16:12.373 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-01T04:16:12.378 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:12.378 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-01T04:16:12.398 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:12.398 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-01T04:16:12.415 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:12.416 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-01T04:16:12.436 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:12.437 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-01T04:16:12.439 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:16:12.439 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:12.439 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 33, 2023-11-01T04:16:12.440 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d", 2023-11-01T04:16:12.440 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2023-11-01T04:15:44.902210+0000", 2023-11-01T04:16:12.440 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2023-11-01T04:16:11.888692+0000", 2023-11-01T04:16:12.440 INFO:teuthology.orchestra.run.smithi153.stdout: "last_up_change": "2023-11-01T04:15:56.634008+0000", 2023-11-01T04:16:12.441 INFO:teuthology.orchestra.run.smithi153.stdout: "last_in_change": "2023-11-01T04:15:49.422762+0000", 2023-11-01T04:16:12.441 INFO:teuthology.orchestra.run.smithi153.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-11-01T04:16:12.441 INFO:teuthology.orchestra.run.smithi153.stdout: "flags_num": 5806080, 2023-11-01T04:16:12.442 INFO:teuthology.orchestra.run.smithi153.stdout: "flags_set": [ 2023-11-01T04:16:12.442 INFO:teuthology.orchestra.run.smithi153.stdout: "nodeep-scrub", 2023-11-01T04:16:12.442 INFO:teuthology.orchestra.run.smithi153.stdout: "noscrub", 2023-11-01T04:16:12.442 INFO:teuthology.orchestra.run.smithi153.stdout: "pglog_hardlimit", 2023-11-01T04:16:12.443 INFO:teuthology.orchestra.run.smithi153.stdout: "purged_snapdirs", 2023-11-01T04:16:12.443 INFO:teuthology.orchestra.run.smithi153.stdout: "recovery_deletes", 2023-11-01T04:16:12.443 INFO:teuthology.orchestra.run.smithi153.stdout: "sortbitwise" 2023-11-01T04:16:12.443 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-11-01T04:16:12.444 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_version": 6, 2023-11-01T04:16:12.444 INFO:teuthology.orchestra.run.smithi153.stdout: "full_ratio": 0.94999998807907104, 2023-11-01T04:16:12.444 INFO:teuthology.orchestra.run.smithi153.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-11-01T04:16:12.444 INFO:teuthology.orchestra.run.smithi153.stdout: "nearfull_ratio": 0.85000002384185791, 2023-11-01T04:16:12.445 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_snapshot": "", 2023-11-01T04:16:12.445 INFO:teuthology.orchestra.run.smithi153.stdout: "pool_max": 3, 2023-11-01T04:16:12.445 INFO:teuthology.orchestra.run.smithi153.stdout: "max_osd": 16, 2023-11-01T04:16:12.445 INFO:teuthology.orchestra.run.smithi153.stdout: "require_min_compat_client": "luminous", 2023-11-01T04:16:12.446 INFO:teuthology.orchestra.run.smithi153.stdout: "min_compat_client": "jewel", 2023-11-01T04:16:12.446 INFO:teuthology.orchestra.run.smithi153.stdout: "require_osd_release": "reef", 2023-11-01T04:16:12.446 INFO:teuthology.orchestra.run.smithi153.stdout: "allow_crimson": false, 2023-11-01T04:16:12.446 INFO:teuthology.orchestra.run.smithi153.stdout: "pools": [ 2023-11-01T04:16:12.447 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.447 INFO:teuthology.orchestra.run.smithi153.stdout: "pool": 1, 2023-11-01T04:16:12.447 INFO:teuthology.orchestra.run.smithi153.stdout: "pool_name": ".mgr", 2023-11-01T04:16:12.447 INFO:teuthology.orchestra.run.smithi153.stdout: "create_time": "2023-11-01T04:15:57.685107+0000", 2023-11-01T04:16:12.448 INFO:teuthology.orchestra.run.smithi153.stdout: "flags": 1, 2023-11-01T04:16:12.448 INFO:teuthology.orchestra.run.smithi153.stdout: "flags_names": "hashpspool", 2023-11-01T04:16:12.448 INFO:teuthology.orchestra.run.smithi153.stdout: "type": 1, 2023-11-01T04:16:12.448 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 2, 2023-11-01T04:16:12.449 INFO:teuthology.orchestra.run.smithi153.stdout: "min_size": 1, 2023-11-01T04:16:12.449 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_rule": 0, 2023-11-01T04:16:12.449 INFO:teuthology.orchestra.run.smithi153.stdout: "peering_crush_bucket_count": 0, 2023-11-01T04:16:12.449 INFO:teuthology.orchestra.run.smithi153.stdout: "peering_crush_bucket_target": 0, 2023-11-01T04:16:12.450 INFO:teuthology.orchestra.run.smithi153.stdout: "peering_crush_bucket_barrier": 0, 2023-11-01T04:16:12.450 INFO:teuthology.orchestra.run.smithi153.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-11-01T04:16:12.450 INFO:teuthology.orchestra.run.smithi153.stdout: "object_hash": 2, 2023-11-01T04:16:12.450 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_autoscale_mode": "off", 2023-11-01T04:16:12.451 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_num": 1, 2023-11-01T04:16:12.451 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_placement_num": 1, 2023-11-01T04:16:12.451 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_placement_num_target": 1, 2023-11-01T04:16:12.451 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_num_target": 1, 2023-11-01T04:16:12.452 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_num_pending": 1, 2023-11-01T04:16:12.452 INFO:teuthology.orchestra.run.smithi153.stdout: "last_pg_merge_meta": { 2023-11-01T04:16:12.452 INFO:teuthology.orchestra.run.smithi153.stdout: "source_pgid": "0.0", 2023-11-01T04:16:12.453 INFO:teuthology.orchestra.run.smithi153.stdout: "ready_epoch": 0, 2023-11-01T04:16:12.453 INFO:teuthology.orchestra.run.smithi153.stdout: "last_epoch_started": 0, 2023-11-01T04:16:12.453 INFO:teuthology.orchestra.run.smithi153.stdout: "last_epoch_clean": 0, 2023-11-01T04:16:12.453 INFO:teuthology.orchestra.run.smithi153.stdout: "source_version": "0'0", 2023-11-01T04:16:12.454 INFO:teuthology.orchestra.run.smithi153.stdout: "target_version": "0'0" 2023-11-01T04:16:12.454 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.454 INFO:teuthology.orchestra.run.smithi153.stdout: "last_change": "25", 2023-11-01T04:16:12.454 INFO:teuthology.orchestra.run.smithi153.stdout: "last_force_op_resend": "0", 2023-11-01T04:16:12.455 INFO:teuthology.orchestra.run.smithi153.stdout: "last_force_op_resend_prenautilus": "0", 2023-11-01T04:16:12.455 INFO:teuthology.orchestra.run.smithi153.stdout: "last_force_op_resend_preluminous": "0", 2023-11-01T04:16:12.455 INFO:teuthology.orchestra.run.smithi153.stdout: "auid": 0, 2023-11-01T04:16:12.455 INFO:teuthology.orchestra.run.smithi153.stdout: "snap_mode": "selfmanaged", 2023-11-01T04:16:12.456 INFO:teuthology.orchestra.run.smithi153.stdout: "snap_seq": 0, 2023-11-01T04:16:12.456 INFO:teuthology.orchestra.run.smithi153.stdout: "snap_epoch": 0, 2023-11-01T04:16:12.456 INFO:teuthology.orchestra.run.smithi153.stdout: "pool_snaps": [], 2023-11-01T04:16:12.456 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_snaps": "[]", 2023-11-01T04:16:12.457 INFO:teuthology.orchestra.run.smithi153.stdout: "quota_max_bytes": 0, 2023-11-01T04:16:12.457 INFO:teuthology.orchestra.run.smithi153.stdout: "quota_max_objects": 0, 2023-11-01T04:16:12.457 INFO:teuthology.orchestra.run.smithi153.stdout: "tiers": [], 2023-11-01T04:16:12.457 INFO:teuthology.orchestra.run.smithi153.stdout: "tier_of": -1, 2023-11-01T04:16:12.458 INFO:teuthology.orchestra.run.smithi153.stdout: "read_tier": -1, 2023-11-01T04:16:12.458 INFO:teuthology.orchestra.run.smithi153.stdout: "write_tier": -1, 2023-11-01T04:16:12.458 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_mode": "none", 2023-11-01T04:16:12.458 INFO:teuthology.orchestra.run.smithi153.stdout: "target_max_bytes": 0, 2023-11-01T04:16:12.459 INFO:teuthology.orchestra.run.smithi153.stdout: "target_max_objects": 0, 2023-11-01T04:16:12.459 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-11-01T04:16:12.459 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-11-01T04:16:12.460 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_target_full_ratio_micro": 800000, 2023-11-01T04:16:12.460 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_min_flush_age": 0, 2023-11-01T04:16:12.460 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_min_evict_age": 0, 2023-11-01T04:16:12.460 INFO:teuthology.orchestra.run.smithi153.stdout: "erasure_code_profile": "", 2023-11-01T04:16:12.461 INFO:teuthology.orchestra.run.smithi153.stdout: "hit_set_params": { 2023-11-01T04:16:12.461 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "none" 2023-11-01T04:16:12.461 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.461 INFO:teuthology.orchestra.run.smithi153.stdout: "hit_set_period": 0, 2023-11-01T04:16:12.462 INFO:teuthology.orchestra.run.smithi153.stdout: "hit_set_count": 0, 2023-11-01T04:16:12.462 INFO:teuthology.orchestra.run.smithi153.stdout: "use_gmt_hitset": true, 2023-11-01T04:16:12.462 INFO:teuthology.orchestra.run.smithi153.stdout: "min_read_recency_for_promote": 0, 2023-11-01T04:16:12.462 INFO:teuthology.orchestra.run.smithi153.stdout: "min_write_recency_for_promote": 0, 2023-11-01T04:16:12.463 INFO:teuthology.orchestra.run.smithi153.stdout: "hit_set_grade_decay_rate": 0, 2023-11-01T04:16:12.463 INFO:teuthology.orchestra.run.smithi153.stdout: "hit_set_search_last_n": 0, 2023-11-01T04:16:12.463 INFO:teuthology.orchestra.run.smithi153.stdout: "grade_table": [], 2023-11-01T04:16:12.463 INFO:teuthology.orchestra.run.smithi153.stdout: "stripe_width": 0, 2023-11-01T04:16:12.464 INFO:teuthology.orchestra.run.smithi153.stdout: "expected_num_objects": 0, 2023-11-01T04:16:12.464 INFO:teuthology.orchestra.run.smithi153.stdout: "fast_read": false, 2023-11-01T04:16:12.464 INFO:teuthology.orchestra.run.smithi153.stdout: "options": { 2023-11-01T04:16:12.464 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_num_max": 32, 2023-11-01T04:16:12.465 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_num_min": 1 2023-11-01T04:16:12.465 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.465 INFO:teuthology.orchestra.run.smithi153.stdout: "application_metadata": { 2023-11-01T04:16:12.465 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": {} 2023-11-01T04:16:12.466 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.466 INFO:teuthology.orchestra.run.smithi153.stdout: "read_balance": { 2023-11-01T04:16:12.466 INFO:teuthology.orchestra.run.smithi153.stdout: "score_acting": 15.380000114440918, 2023-11-01T04:16:12.466 INFO:teuthology.orchestra.run.smithi153.stdout: "score_stable": 15.380000114440918, 2023-11-01T04:16:12.466 INFO:teuthology.orchestra.run.smithi153.stdout: "optimal_score": 0.12999999523162842, 2023-11-01T04:16:12.467 INFO:teuthology.orchestra.run.smithi153.stdout: "raw_score_acting": 2, 2023-11-01T04:16:12.467 INFO:teuthology.orchestra.run.smithi153.stdout: "raw_score_stable": 2, 2023-11-01T04:16:12.467 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity_weighted": 1, 2023-11-01T04:16:12.467 INFO:teuthology.orchestra.run.smithi153.stdout: "average_primary_affinity": 1, 2023-11-01T04:16:12.468 INFO:teuthology.orchestra.run.smithi153.stdout: "average_primary_affinity_weighted": 1 2023-11-01T04:16:12.468 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.468 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.468 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.469 INFO:teuthology.orchestra.run.smithi153.stdout: "pool": 2, 2023-11-01T04:16:12.469 INFO:teuthology.orchestra.run.smithi153.stdout: "pool_name": "rbd", 2023-11-01T04:16:12.469 INFO:teuthology.orchestra.run.smithi153.stdout: "create_time": "2023-11-01T04:15:59.882949+0000", 2023-11-01T04:16:12.469 INFO:teuthology.orchestra.run.smithi153.stdout: "flags": 1, 2023-11-01T04:16:12.470 INFO:teuthology.orchestra.run.smithi153.stdout: "flags_names": "hashpspool", 2023-11-01T04:16:12.470 INFO:teuthology.orchestra.run.smithi153.stdout: "type": 1, 2023-11-01T04:16:12.470 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 2, 2023-11-01T04:16:12.470 INFO:teuthology.orchestra.run.smithi153.stdout: "min_size": 1, 2023-11-01T04:16:12.471 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_rule": 0, 2023-11-01T04:16:12.471 INFO:teuthology.orchestra.run.smithi153.stdout: "peering_crush_bucket_count": 0, 2023-11-01T04:16:12.471 INFO:teuthology.orchestra.run.smithi153.stdout: "peering_crush_bucket_target": 0, 2023-11-01T04:16:12.471 INFO:teuthology.orchestra.run.smithi153.stdout: "peering_crush_bucket_barrier": 0, 2023-11-01T04:16:12.472 INFO:teuthology.orchestra.run.smithi153.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-11-01T04:16:12.472 INFO:teuthology.orchestra.run.smithi153.stdout: "object_hash": 2, 2023-11-01T04:16:12.472 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_autoscale_mode": "off", 2023-11-01T04:16:12.472 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_num": 8, 2023-11-01T04:16:12.473 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_placement_num": 8, 2023-11-01T04:16:12.473 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_placement_num_target": 8, 2023-11-01T04:16:12.473 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_num_target": 8, 2023-11-01T04:16:12.473 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_num_pending": 8, 2023-11-01T04:16:12.474 INFO:teuthology.orchestra.run.smithi153.stdout: "last_pg_merge_meta": { 2023-11-01T04:16:12.474 INFO:teuthology.orchestra.run.smithi153.stdout: "source_pgid": "0.0", 2023-11-01T04:16:12.474 INFO:teuthology.orchestra.run.smithi153.stdout: "ready_epoch": 0, 2023-11-01T04:16:12.474 INFO:teuthology.orchestra.run.smithi153.stdout: "last_epoch_started": 0, 2023-11-01T04:16:12.475 INFO:teuthology.orchestra.run.smithi153.stdout: "last_epoch_clean": 0, 2023-11-01T04:16:12.475 INFO:teuthology.orchestra.run.smithi153.stdout: "source_version": "0'0", 2023-11-01T04:16:12.475 INFO:teuthology.orchestra.run.smithi153.stdout: "target_version": "0'0" 2023-11-01T04:16:12.475 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.475 INFO:teuthology.orchestra.run.smithi153.stdout: "last_change": "28", 2023-11-01T04:16:12.476 INFO:teuthology.orchestra.run.smithi153.stdout: "last_force_op_resend": "0", 2023-11-01T04:16:12.476 INFO:teuthology.orchestra.run.smithi153.stdout: "last_force_op_resend_prenautilus": "0", 2023-11-01T04:16:12.476 INFO:teuthology.orchestra.run.smithi153.stdout: "last_force_op_resend_preluminous": "0", 2023-11-01T04:16:12.476 INFO:teuthology.orchestra.run.smithi153.stdout: "auid": 0, 2023-11-01T04:16:12.477 INFO:teuthology.orchestra.run.smithi153.stdout: "snap_mode": "selfmanaged", 2023-11-01T04:16:12.477 INFO:teuthology.orchestra.run.smithi153.stdout: "snap_seq": 0, 2023-11-01T04:16:12.477 INFO:teuthology.orchestra.run.smithi153.stdout: "snap_epoch": 0, 2023-11-01T04:16:12.477 INFO:teuthology.orchestra.run.smithi153.stdout: "pool_snaps": [], 2023-11-01T04:16:12.477 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_snaps": "[]", 2023-11-01T04:16:12.478 INFO:teuthology.orchestra.run.smithi153.stdout: "quota_max_bytes": 0, 2023-11-01T04:16:12.478 INFO:teuthology.orchestra.run.smithi153.stdout: "quota_max_objects": 0, 2023-11-01T04:16:12.478 INFO:teuthology.orchestra.run.smithi153.stdout: "tiers": [], 2023-11-01T04:16:12.478 INFO:teuthology.orchestra.run.smithi153.stdout: "tier_of": -1, 2023-11-01T04:16:12.479 INFO:teuthology.orchestra.run.smithi153.stdout: "read_tier": -1, 2023-11-01T04:16:12.479 INFO:teuthology.orchestra.run.smithi153.stdout: "write_tier": -1, 2023-11-01T04:16:12.479 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_mode": "none", 2023-11-01T04:16:12.479 INFO:teuthology.orchestra.run.smithi153.stdout: "target_max_bytes": 0, 2023-11-01T04:16:12.480 INFO:teuthology.orchestra.run.smithi153.stdout: "target_max_objects": 0, 2023-11-01T04:16:12.480 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-11-01T04:16:12.480 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-11-01T04:16:12.480 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_target_full_ratio_micro": 800000, 2023-11-01T04:16:12.481 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_min_flush_age": 0, 2023-11-01T04:16:12.481 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_min_evict_age": 0, 2023-11-01T04:16:12.481 INFO:teuthology.orchestra.run.smithi153.stdout: "erasure_code_profile": "", 2023-11-01T04:16:12.481 INFO:teuthology.orchestra.run.smithi153.stdout: "hit_set_params": { 2023-11-01T04:16:12.481 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "none" 2023-11-01T04:16:12.482 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.482 INFO:teuthology.orchestra.run.smithi153.stdout: "hit_set_period": 0, 2023-11-01T04:16:12.482 INFO:teuthology.orchestra.run.smithi153.stdout: "hit_set_count": 0, 2023-11-01T04:16:12.482 INFO:teuthology.orchestra.run.smithi153.stdout: "use_gmt_hitset": true, 2023-11-01T04:16:12.483 INFO:teuthology.orchestra.run.smithi153.stdout: "min_read_recency_for_promote": 0, 2023-11-01T04:16:12.483 INFO:teuthology.orchestra.run.smithi153.stdout: "min_write_recency_for_promote": 0, 2023-11-01T04:16:12.483 INFO:teuthology.orchestra.run.smithi153.stdout: "hit_set_grade_decay_rate": 0, 2023-11-01T04:16:12.483 INFO:teuthology.orchestra.run.smithi153.stdout: "hit_set_search_last_n": 0, 2023-11-01T04:16:12.484 INFO:teuthology.orchestra.run.smithi153.stdout: "grade_table": [], 2023-11-01T04:16:12.484 INFO:teuthology.orchestra.run.smithi153.stdout: "stripe_width": 0, 2023-11-01T04:16:12.484 INFO:teuthology.orchestra.run.smithi153.stdout: "expected_num_objects": 0, 2023-11-01T04:16:12.484 INFO:teuthology.orchestra.run.smithi153.stdout: "fast_read": false, 2023-11-01T04:16:12.484 INFO:teuthology.orchestra.run.smithi153.stdout: "options": {}, 2023-11-01T04:16:12.485 INFO:teuthology.orchestra.run.smithi153.stdout: "application_metadata": { 2023-11-01T04:16:12.485 INFO:teuthology.orchestra.run.smithi153.stdout: "rbd": {} 2023-11-01T04:16:12.485 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.485 INFO:teuthology.orchestra.run.smithi153.stdout: "read_balance": { 2023-11-01T04:16:12.485 INFO:teuthology.orchestra.run.smithi153.stdout: "score_acting": 5.9899997711181641, 2023-11-01T04:16:12.486 INFO:teuthology.orchestra.run.smithi153.stdout: "score_stable": 5.9899997711181641, 2023-11-01T04:16:12.486 INFO:teuthology.orchestra.run.smithi153.stdout: "optimal_score": 0.68999999761581421, 2023-11-01T04:16:12.486 INFO:teuthology.orchestra.run.smithi153.stdout: "raw_score_acting": 4.130000114440918, 2023-11-01T04:16:12.486 INFO:teuthology.orchestra.run.smithi153.stdout: "raw_score_stable": 4.130000114440918, 2023-11-01T04:16:12.487 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity_weighted": 1, 2023-11-01T04:16:12.487 INFO:teuthology.orchestra.run.smithi153.stdout: "average_primary_affinity": 1, 2023-11-01T04:16:12.487 INFO:teuthology.orchestra.run.smithi153.stdout: "average_primary_affinity_weighted": 1 2023-11-01T04:16:12.487 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.487 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.488 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.488 INFO:teuthology.orchestra.run.smithi153.stdout: "pool": 3, 2023-11-01T04:16:12.488 INFO:teuthology.orchestra.run.smithi153.stdout: "pool_name": "unique_pool_0", 2023-11-01T04:16:12.488 INFO:teuthology.orchestra.run.smithi153.stdout: "create_time": "2023-11-01T04:16:08.908876+0000", 2023-11-01T04:16:12.489 INFO:teuthology.orchestra.run.smithi153.stdout: "flags": 1, 2023-11-01T04:16:12.489 INFO:teuthology.orchestra.run.smithi153.stdout: "flags_names": "hashpspool", 2023-11-01T04:16:12.489 INFO:teuthology.orchestra.run.smithi153.stdout: "type": 3, 2023-11-01T04:16:12.489 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 7, 2023-11-01T04:16:12.489 INFO:teuthology.orchestra.run.smithi153.stdout: "min_size": 5, 2023-11-01T04:16:12.490 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_rule": 1, 2023-11-01T04:16:12.490 INFO:teuthology.orchestra.run.smithi153.stdout: "peering_crush_bucket_count": 0, 2023-11-01T04:16:12.490 INFO:teuthology.orchestra.run.smithi153.stdout: "peering_crush_bucket_target": 0, 2023-11-01T04:16:12.490 INFO:teuthology.orchestra.run.smithi153.stdout: "peering_crush_bucket_barrier": 0, 2023-11-01T04:16:12.491 INFO:teuthology.orchestra.run.smithi153.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-11-01T04:16:12.491 INFO:teuthology.orchestra.run.smithi153.stdout: "object_hash": 2, 2023-11-01T04:16:12.491 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_autoscale_mode": "off", 2023-11-01T04:16:12.491 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_num": 16, 2023-11-01T04:16:12.491 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_placement_num": 16, 2023-11-01T04:16:12.492 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_placement_num_target": 16, 2023-11-01T04:16:12.492 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_num_target": 16, 2023-11-01T04:16:12.492 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_num_pending": 16, 2023-11-01T04:16:12.492 INFO:teuthology.orchestra.run.smithi153.stdout: "last_pg_merge_meta": { 2023-11-01T04:16:12.493 INFO:teuthology.orchestra.run.smithi153.stdout: "source_pgid": "0.0", 2023-11-01T04:16:12.493 INFO:teuthology.orchestra.run.smithi153.stdout: "ready_epoch": 0, 2023-11-01T04:16:12.493 INFO:teuthology.orchestra.run.smithi153.stdout: "last_epoch_started": 0, 2023-11-01T04:16:12.493 INFO:teuthology.orchestra.run.smithi153.stdout: "last_epoch_clean": 0, 2023-11-01T04:16:12.493 INFO:teuthology.orchestra.run.smithi153.stdout: "source_version": "0'0", 2023-11-01T04:16:12.494 INFO:teuthology.orchestra.run.smithi153.stdout: "target_version": "0'0" 2023-11-01T04:16:12.494 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.494 INFO:teuthology.orchestra.run.smithi153.stdout: "last_change": "33", 2023-11-01T04:16:12.494 INFO:teuthology.orchestra.run.smithi153.stdout: "last_force_op_resend": "0", 2023-11-01T04:16:12.494 INFO:teuthology.orchestra.run.smithi153.stdout: "last_force_op_resend_prenautilus": "0", 2023-11-01T04:16:12.495 INFO:teuthology.orchestra.run.smithi153.stdout: "last_force_op_resend_preluminous": "0", 2023-11-01T04:16:12.495 INFO:teuthology.orchestra.run.smithi153.stdout: "auid": 0, 2023-11-01T04:16:12.495 INFO:teuthology.orchestra.run.smithi153.stdout: "snap_mode": "selfmanaged", 2023-11-01T04:16:12.495 INFO:teuthology.orchestra.run.smithi153.stdout: "snap_seq": 0, 2023-11-01T04:16:12.495 INFO:teuthology.orchestra.run.smithi153.stdout: "snap_epoch": 0, 2023-11-01T04:16:12.496 INFO:teuthology.orchestra.run.smithi153.stdout: "pool_snaps": [], 2023-11-01T04:16:12.496 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_snaps": "[]", 2023-11-01T04:16:12.496 INFO:teuthology.orchestra.run.smithi153.stdout: "quota_max_bytes": 0, 2023-11-01T04:16:12.496 INFO:teuthology.orchestra.run.smithi153.stdout: "quota_max_objects": 0, 2023-11-01T04:16:12.496 INFO:teuthology.orchestra.run.smithi153.stdout: "tiers": [], 2023-11-01T04:16:12.497 INFO:teuthology.orchestra.run.smithi153.stdout: "tier_of": -1, 2023-11-01T04:16:12.497 INFO:teuthology.orchestra.run.smithi153.stdout: "read_tier": -1, 2023-11-01T04:16:12.497 INFO:teuthology.orchestra.run.smithi153.stdout: "write_tier": -1, 2023-11-01T04:16:12.497 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_mode": "none", 2023-11-01T04:16:12.498 INFO:teuthology.orchestra.run.smithi153.stdout: "target_max_bytes": 0, 2023-11-01T04:16:12.498 INFO:teuthology.orchestra.run.smithi153.stdout: "target_max_objects": 0, 2023-11-01T04:16:12.498 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-11-01T04:16:12.498 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-11-01T04:16:12.498 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_target_full_ratio_micro": 800000, 2023-11-01T04:16:12.499 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_min_flush_age": 0, 2023-11-01T04:16:12.499 INFO:teuthology.orchestra.run.smithi153.stdout: "cache_min_evict_age": 0, 2023-11-01T04:16:12.499 INFO:teuthology.orchestra.run.smithi153.stdout: "erasure_code_profile": "shecprofile", 2023-11-01T04:16:12.499 INFO:teuthology.orchestra.run.smithi153.stdout: "hit_set_params": { 2023-11-01T04:16:12.500 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "none" 2023-11-01T04:16:12.500 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.500 INFO:teuthology.orchestra.run.smithi153.stdout: "hit_set_period": 0, 2023-11-01T04:16:12.500 INFO:teuthology.orchestra.run.smithi153.stdout: "hit_set_count": 0, 2023-11-01T04:16:12.501 INFO:teuthology.orchestra.run.smithi153.stdout: "use_gmt_hitset": true, 2023-11-01T04:16:12.501 INFO:teuthology.orchestra.run.smithi153.stdout: "min_read_recency_for_promote": 0, 2023-11-01T04:16:12.501 INFO:teuthology.orchestra.run.smithi153.stdout: "min_write_recency_for_promote": 0, 2023-11-01T04:16:12.501 INFO:teuthology.orchestra.run.smithi153.stdout: "hit_set_grade_decay_rate": 0, 2023-11-01T04:16:12.502 INFO:teuthology.orchestra.run.smithi153.stdout: "hit_set_search_last_n": 0, 2023-11-01T04:16:12.502 INFO:teuthology.orchestra.run.smithi153.stdout: "grade_table": [], 2023-11-01T04:16:12.502 INFO:teuthology.orchestra.run.smithi153.stdout: "stripe_width": 16384, 2023-11-01T04:16:12.502 INFO:teuthology.orchestra.run.smithi153.stdout: "expected_num_objects": 0, 2023-11-01T04:16:12.502 INFO:teuthology.orchestra.run.smithi153.stdout: "fast_read": false, 2023-11-01T04:16:12.503 INFO:teuthology.orchestra.run.smithi153.stdout: "options": {}, 2023-11-01T04:16:12.503 INFO:teuthology.orchestra.run.smithi153.stdout: "application_metadata": { 2023-11-01T04:16:12.503 INFO:teuthology.orchestra.run.smithi153.stdout: "rados": {} 2023-11-01T04:16:12.503 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.504 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.504 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-11-01T04:16:12.504 INFO:teuthology.orchestra.run.smithi153.stdout: "osds": [ 2023-11-01T04:16:12.504 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.505 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 0, 2023-11-01T04:16:12.505 INFO:teuthology.orchestra.run.smithi153.stdout: "uuid": "83ce3480-defe-4c73-aa1d-47c7941a23ba", 2023-11-01T04:16:12.505 INFO:teuthology.orchestra.run.smithi153.stdout: "up": 1, 2023-11-01T04:16:12.505 INFO:teuthology.orchestra.run.smithi153.stdout: "in": 1, 2023-11-01T04:16:12.505 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 1, 2023-11-01T04:16:12.506 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity": 1, 2023-11-01T04:16:12.506 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_begin": 0, 2023-11-01T04:16:12.506 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_end": 0, 2023-11-01T04:16:12.506 INFO:teuthology.orchestra.run.smithi153.stdout: "up_from": 22, 2023-11-01T04:16:12.507 INFO:teuthology.orchestra.run.smithi153.stdout: "up_thru": 31, 2023-11-01T04:16:12.507 INFO:teuthology.orchestra.run.smithi153.stdout: "down_at": 0, 2023-11-01T04:16:12.507 INFO:teuthology.orchestra.run.smithi153.stdout: "lost_at": 0, 2023-11-01T04:16:12.507 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-11-01T04:16:12.508 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.508 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.508 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.508 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6816", 2023-11-01T04:16:12.509 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 59702962 2023-11-01T04:16:12.509 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.509 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.509 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.509 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6817", 2023-11-01T04:16:12.510 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 59702962 2023-11-01T04:16:12.510 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.510 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.510 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.511 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addrs": { 2023-11-01T04:16:12.511 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.511 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.511 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.512 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6818", 2023-11-01T04:16:12.512 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 59702962 2023-11-01T04:16:12.512 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.512 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.512 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.513 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6819", 2023-11-01T04:16:12.513 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 59702962 2023-11-01T04:16:12.513 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.513 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.514 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.514 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addrs": { 2023-11-01T04:16:12.514 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.514 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.514 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.515 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6822", 2023-11-01T04:16:12.515 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 59702962 2023-11-01T04:16:12.515 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.515 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.515 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.516 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6823", 2023-11-01T04:16:12.516 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 59702962 2023-11-01T04:16:12.516 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.516 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.516 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.517 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addrs": { 2023-11-01T04:16:12.517 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.517 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.517 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.517 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6820", 2023-11-01T04:16:12.518 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 59702962 2023-11-01T04:16:12.518 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.518 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.518 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.518 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6821", 2023-11-01T04:16:12.519 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 59702962 2023-11-01T04:16:12.519 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.519 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.519 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.519 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6817/59702962", 2023-11-01T04:16:12.520 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "172.21.15.153:6819/59702962", 2023-11-01T04:16:12.520 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addr": "172.21.15.153:6823/59702962", 2023-11-01T04:16:12.520 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addr": "172.21.15.153:6821/59702962", 2023-11-01T04:16:12.520 INFO:teuthology.orchestra.run.smithi153.stdout: "state": [ 2023-11-01T04:16:12.520 INFO:teuthology.orchestra.run.smithi153.stdout: "exists", 2023-11-01T04:16:12.521 INFO:teuthology.orchestra.run.smithi153.stdout: "up" 2023-11-01T04:16:12.521 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.521 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.521 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.522 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 1, 2023-11-01T04:16:12.522 INFO:teuthology.orchestra.run.smithi153.stdout: "uuid": "01a9801f-72fb-4299-8708-d70eb59b0fb2", 2023-11-01T04:16:12.522 INFO:teuthology.orchestra.run.smithi153.stdout: "up": 1, 2023-11-01T04:16:12.522 INFO:teuthology.orchestra.run.smithi153.stdout: "in": 1, 2023-11-01T04:16:12.522 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 1, 2023-11-01T04:16:12.523 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity": 1, 2023-11-01T04:16:12.523 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_begin": 0, 2023-11-01T04:16:12.523 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_end": 0, 2023-11-01T04:16:12.523 INFO:teuthology.orchestra.run.smithi153.stdout: "up_from": 22, 2023-11-01T04:16:12.523 INFO:teuthology.orchestra.run.smithi153.stdout: "up_thru": 0, 2023-11-01T04:16:12.523 INFO:teuthology.orchestra.run.smithi153.stdout: "down_at": 0, 2023-11-01T04:16:12.524 INFO:teuthology.orchestra.run.smithi153.stdout: "lost_at": 0, 2023-11-01T04:16:12.524 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-11-01T04:16:12.524 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.524 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.524 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.525 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6813", 2023-11-01T04:16:12.525 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3103315277 2023-11-01T04:16:12.525 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.525 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.525 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.526 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6815", 2023-11-01T04:16:12.526 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3103315277 2023-11-01T04:16:12.526 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.526 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.526 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.527 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addrs": { 2023-11-01T04:16:12.527 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.527 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.527 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.527 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6817", 2023-11-01T04:16:12.527 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3103315277 2023-11-01T04:16:12.528 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.528 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.528 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.528 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6818", 2023-11-01T04:16:12.528 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3103315277 2023-11-01T04:16:12.529 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.529 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.529 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.529 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addrs": { 2023-11-01T04:16:12.529 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.530 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.530 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.530 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6822", 2023-11-01T04:16:12.530 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3103315277 2023-11-01T04:16:12.531 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.531 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.531 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.531 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6823", 2023-11-01T04:16:12.531 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3103315277 2023-11-01T04:16:12.532 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.532 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.532 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.532 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addrs": { 2023-11-01T04:16:12.532 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.532 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.533 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.533 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6820", 2023-11-01T04:16:12.533 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3103315277 2023-11-01T04:16:12.533 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.533 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.534 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.534 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6821", 2023-11-01T04:16:12.534 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3103315277 2023-11-01T04:16:12.534 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.534 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.535 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.535 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.173:6815/3103315277", 2023-11-01T04:16:12.535 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "172.21.15.173:6818/3103315277", 2023-11-01T04:16:12.535 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addr": "172.21.15.173:6823/3103315277", 2023-11-01T04:16:12.535 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addr": "172.21.15.173:6821/3103315277", 2023-11-01T04:16:12.536 INFO:teuthology.orchestra.run.smithi153.stdout: "state": [ 2023-11-01T04:16:12.536 INFO:teuthology.orchestra.run.smithi153.stdout: "exists", 2023-11-01T04:16:12.536 INFO:teuthology.orchestra.run.smithi153.stdout: "up" 2023-11-01T04:16:12.536 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.536 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.536 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.537 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 2, 2023-11-01T04:16:12.537 INFO:teuthology.orchestra.run.smithi153.stdout: "uuid": "7c16db19-9bec-4ca0-88eb-bb5fb673d160", 2023-11-01T04:16:12.537 INFO:teuthology.orchestra.run.smithi153.stdout: "up": 1, 2023-11-01T04:16:12.537 INFO:teuthology.orchestra.run.smithi153.stdout: "in": 1, 2023-11-01T04:16:12.537 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 1, 2023-11-01T04:16:12.538 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity": 1, 2023-11-01T04:16:12.538 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_begin": 0, 2023-11-01T04:16:12.538 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_end": 0, 2023-11-01T04:16:12.538 INFO:teuthology.orchestra.run.smithi153.stdout: "up_from": 22, 2023-11-01T04:16:12.538 INFO:teuthology.orchestra.run.smithi153.stdout: "up_thru": 0, 2023-11-01T04:16:12.539 INFO:teuthology.orchestra.run.smithi153.stdout: "down_at": 0, 2023-11-01T04:16:12.539 INFO:teuthology.orchestra.run.smithi153.stdout: "lost_at": 0, 2023-11-01T04:16:12.539 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-11-01T04:16:12.539 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.539 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.540 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.540 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6816", 2023-11-01T04:16:12.540 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1072240131 2023-11-01T04:16:12.540 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.540 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.541 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.541 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6817", 2023-11-01T04:16:12.541 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1072240131 2023-11-01T04:16:12.541 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.541 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.542 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.542 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addrs": { 2023-11-01T04:16:12.542 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.542 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.542 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.542 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6818", 2023-11-01T04:16:12.543 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1072240131 2023-11-01T04:16:12.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.543 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.543 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.543 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6820", 2023-11-01T04:16:12.544 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1072240131 2023-11-01T04:16:12.544 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.544 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.544 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addrs": { 2023-11-01T04:16:12.545 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.545 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.545 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.545 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6824", 2023-11-01T04:16:12.545 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1072240131 2023-11-01T04:16:12.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.546 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.546 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.546 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6825", 2023-11-01T04:16:12.546 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1072240131 2023-11-01T04:16:12.547 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.547 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.547 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.547 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addrs": { 2023-11-01T04:16:12.547 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.548 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.548 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.548 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6821", 2023-11-01T04:16:12.548 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1072240131 2023-11-01T04:16:12.549 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.549 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.549 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.549 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6823", 2023-11-01T04:16:12.550 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1072240131 2023-11-01T04:16:12.550 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.550 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.550 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6817/1072240131", 2023-11-01T04:16:12.551 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "172.21.15.181:6820/1072240131", 2023-11-01T04:16:12.551 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addr": "172.21.15.181:6825/1072240131", 2023-11-01T04:16:12.551 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addr": "172.21.15.181:6823/1072240131", 2023-11-01T04:16:12.551 INFO:teuthology.orchestra.run.smithi153.stdout: "state": [ 2023-11-01T04:16:12.551 INFO:teuthology.orchestra.run.smithi153.stdout: "exists", 2023-11-01T04:16:12.552 INFO:teuthology.orchestra.run.smithi153.stdout: "up" 2023-11-01T04:16:12.552 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.552 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.552 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.552 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 3, 2023-11-01T04:16:12.553 INFO:teuthology.orchestra.run.smithi153.stdout: "uuid": "4c4cc8d1-6761-4d20-90e8-7c3d43d889d8", 2023-11-01T04:16:12.553 INFO:teuthology.orchestra.run.smithi153.stdout: "up": 1, 2023-11-01T04:16:12.553 INFO:teuthology.orchestra.run.smithi153.stdout: "in": 1, 2023-11-01T04:16:12.553 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 1, 2023-11-01T04:16:12.553 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity": 1, 2023-11-01T04:16:12.554 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_begin": 0, 2023-11-01T04:16:12.554 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_end": 0, 2023-11-01T04:16:12.554 INFO:teuthology.orchestra.run.smithi153.stdout: "up_from": 22, 2023-11-01T04:16:12.554 INFO:teuthology.orchestra.run.smithi153.stdout: "up_thru": 31, 2023-11-01T04:16:12.554 INFO:teuthology.orchestra.run.smithi153.stdout: "down_at": 0, 2023-11-01T04:16:12.555 INFO:teuthology.orchestra.run.smithi153.stdout: "lost_at": 0, 2023-11-01T04:16:12.555 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-11-01T04:16:12.555 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.555 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.555 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.556 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6826", 2023-11-01T04:16:12.556 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2304315422 2023-11-01T04:16:12.556 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.556 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.556 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.557 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6827", 2023-11-01T04:16:12.557 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2304315422 2023-11-01T04:16:12.557 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.557 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.557 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.557 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addrs": { 2023-11-01T04:16:12.558 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.558 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.558 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.558 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6828", 2023-11-01T04:16:12.558 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2304315422 2023-11-01T04:16:12.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.559 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.559 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.559 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6829", 2023-11-01T04:16:12.559 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2304315422 2023-11-01T04:16:12.560 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.560 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.560 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.560 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addrs": { 2023-11-01T04:16:12.560 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.561 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.561 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.561 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6832", 2023-11-01T04:16:12.561 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2304315422 2023-11-01T04:16:12.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.561 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.562 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.562 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6833", 2023-11-01T04:16:12.562 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2304315422 2023-11-01T04:16:12.562 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.562 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.563 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addrs": { 2023-11-01T04:16:12.563 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.563 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.563 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.563 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6830", 2023-11-01T04:16:12.564 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2304315422 2023-11-01T04:16:12.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.564 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.564 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.564 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6831", 2023-11-01T04:16:12.564 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2304315422 2023-11-01T04:16:12.565 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.565 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.565 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.190:6827/2304315422", 2023-11-01T04:16:12.565 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "172.21.15.190:6829/2304315422", 2023-11-01T04:16:12.566 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addr": "172.21.15.190:6833/2304315422", 2023-11-01T04:16:12.566 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addr": "172.21.15.190:6831/2304315422", 2023-11-01T04:16:12.566 INFO:teuthology.orchestra.run.smithi153.stdout: "state": [ 2023-11-01T04:16:12.566 INFO:teuthology.orchestra.run.smithi153.stdout: "exists", 2023-11-01T04:16:12.566 INFO:teuthology.orchestra.run.smithi153.stdout: "up" 2023-11-01T04:16:12.566 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.567 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.567 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 4, 2023-11-01T04:16:12.567 INFO:teuthology.orchestra.run.smithi153.stdout: "uuid": "0fb14630-87e6-4456-9640-7986a5079821", 2023-11-01T04:16:12.567 INFO:teuthology.orchestra.run.smithi153.stdout: "up": 1, 2023-11-01T04:16:12.568 INFO:teuthology.orchestra.run.smithi153.stdout: "in": 1, 2023-11-01T04:16:12.568 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 1, 2023-11-01T04:16:12.568 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity": 1, 2023-11-01T04:16:12.568 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_begin": 0, 2023-11-01T04:16:12.568 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_end": 0, 2023-11-01T04:16:12.568 INFO:teuthology.orchestra.run.smithi153.stdout: "up_from": 22, 2023-11-01T04:16:12.569 INFO:teuthology.orchestra.run.smithi153.stdout: "up_thru": 0, 2023-11-01T04:16:12.569 INFO:teuthology.orchestra.run.smithi153.stdout: "down_at": 0, 2023-11-01T04:16:12.569 INFO:teuthology.orchestra.run.smithi153.stdout: "lost_at": 0, 2023-11-01T04:16:12.569 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-11-01T04:16:12.569 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.570 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.570 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.570 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6824", 2023-11-01T04:16:12.570 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3209586931 2023-11-01T04:16:12.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.571 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.571 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.571 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6825", 2023-11-01T04:16:12.571 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3209586931 2023-11-01T04:16:12.571 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.572 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.572 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.572 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addrs": { 2023-11-01T04:16:12.572 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.572 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.573 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.573 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6826", 2023-11-01T04:16:12.573 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3209586931 2023-11-01T04:16:12.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.573 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.574 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.574 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6827", 2023-11-01T04:16:12.574 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3209586931 2023-11-01T04:16:12.574 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.574 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.575 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addrs": { 2023-11-01T04:16:12.575 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.575 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.575 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.576 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6830", 2023-11-01T04:16:12.576 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3209586931 2023-11-01T04:16:12.576 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.576 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.576 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.577 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6831", 2023-11-01T04:16:12.577 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3209586931 2023-11-01T04:16:12.577 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.577 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.577 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.578 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addrs": { 2023-11-01T04:16:12.578 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.578 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.578 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.578 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6828", 2023-11-01T04:16:12.579 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3209586931 2023-11-01T04:16:12.579 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.579 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.579 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.579 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6829", 2023-11-01T04:16:12.579 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3209586931 2023-11-01T04:16:12.580 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.580 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.580 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.580 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6825/3209586931", 2023-11-01T04:16:12.580 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "172.21.15.153:6827/3209586931", 2023-11-01T04:16:12.581 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addr": "172.21.15.153:6831/3209586931", 2023-11-01T04:16:12.581 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addr": "172.21.15.153:6829/3209586931", 2023-11-01T04:16:12.581 INFO:teuthology.orchestra.run.smithi153.stdout: "state": [ 2023-11-01T04:16:12.581 INFO:teuthology.orchestra.run.smithi153.stdout: "exists", 2023-11-01T04:16:12.581 INFO:teuthology.orchestra.run.smithi153.stdout: "up" 2023-11-01T04:16:12.582 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.582 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.582 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.582 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 5, 2023-11-01T04:16:12.582 INFO:teuthology.orchestra.run.smithi153.stdout: "uuid": "cc715599-0461-477b-beb2-8777d6f36b22", 2023-11-01T04:16:12.582 INFO:teuthology.orchestra.run.smithi153.stdout: "up": 1, 2023-11-01T04:16:12.583 INFO:teuthology.orchestra.run.smithi153.stdout: "in": 1, 2023-11-01T04:16:12.583 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 1, 2023-11-01T04:16:12.583 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity": 1, 2023-11-01T04:16:12.583 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_begin": 0, 2023-11-01T04:16:12.583 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_end": 0, 2023-11-01T04:16:12.583 INFO:teuthology.orchestra.run.smithi153.stdout: "up_from": 22, 2023-11-01T04:16:12.584 INFO:teuthology.orchestra.run.smithi153.stdout: "up_thru": 31, 2023-11-01T04:16:12.584 INFO:teuthology.orchestra.run.smithi153.stdout: "down_at": 0, 2023-11-01T04:16:12.584 INFO:teuthology.orchestra.run.smithi153.stdout: "lost_at": 0, 2023-11-01T04:16:12.584 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-11-01T04:16:12.584 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.585 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.585 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.585 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6824", 2023-11-01T04:16:12.585 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 409879231 2023-11-01T04:16:12.585 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.585 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.586 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.586 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6825", 2023-11-01T04:16:12.586 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 409879231 2023-11-01T04:16:12.586 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.586 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.587 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addrs": { 2023-11-01T04:16:12.587 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.587 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.587 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.587 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6826", 2023-11-01T04:16:12.587 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 409879231 2023-11-01T04:16:12.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.588 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.588 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.588 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6827", 2023-11-01T04:16:12.588 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 409879231 2023-11-01T04:16:12.588 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.589 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.589 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addrs": { 2023-11-01T04:16:12.589 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.589 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.590 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.590 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6830", 2023-11-01T04:16:12.590 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 409879231 2023-11-01T04:16:12.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.590 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.590 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.591 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6831", 2023-11-01T04:16:12.591 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 409879231 2023-11-01T04:16:12.591 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.591 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.593 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addrs": { 2023-11-01T04:16:12.593 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.593 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.593 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.594 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6828", 2023-11-01T04:16:12.594 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 409879231 2023-11-01T04:16:12.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.594 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.594 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.595 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6829", 2023-11-01T04:16:12.595 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 409879231 2023-11-01T04:16:12.595 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.595 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.595 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.173:6825/409879231", 2023-11-01T04:16:12.596 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "172.21.15.173:6827/409879231", 2023-11-01T04:16:12.596 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addr": "172.21.15.173:6831/409879231", 2023-11-01T04:16:12.596 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addr": "172.21.15.173:6829/409879231", 2023-11-01T04:16:12.596 INFO:teuthology.orchestra.run.smithi153.stdout: "state": [ 2023-11-01T04:16:12.596 INFO:teuthology.orchestra.run.smithi153.stdout: "exists", 2023-11-01T04:16:12.596 INFO:teuthology.orchestra.run.smithi153.stdout: "up" 2023-11-01T04:16:12.597 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.597 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.597 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.597 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 6, 2023-11-01T04:16:12.597 INFO:teuthology.orchestra.run.smithi153.stdout: "uuid": "ca5bea1c-f309-4258-b85d-738097037962", 2023-11-01T04:16:12.597 INFO:teuthology.orchestra.run.smithi153.stdout: "up": 1, 2023-11-01T04:16:12.598 INFO:teuthology.orchestra.run.smithi153.stdout: "in": 1, 2023-11-01T04:16:12.598 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 1, 2023-11-01T04:16:12.598 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity": 1, 2023-11-01T04:16:12.598 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_begin": 0, 2023-11-01T04:16:12.598 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_end": 0, 2023-11-01T04:16:12.599 INFO:teuthology.orchestra.run.smithi153.stdout: "up_from": 21, 2023-11-01T04:16:12.599 INFO:teuthology.orchestra.run.smithi153.stdout: "up_thru": 31, 2023-11-01T04:16:12.599 INFO:teuthology.orchestra.run.smithi153.stdout: "down_at": 0, 2023-11-01T04:16:12.599 INFO:teuthology.orchestra.run.smithi153.stdout: "lost_at": 0, 2023-11-01T04:16:12.599 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-11-01T04:16:12.599 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.600 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.600 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.600 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6819", 2023-11-01T04:16:12.600 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3465541770 2023-11-01T04:16:12.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.600 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.601 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.601 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6822", 2023-11-01T04:16:12.601 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3465541770 2023-11-01T04:16:12.601 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.601 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.602 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.602 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addrs": { 2023-11-01T04:16:12.602 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.602 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.602 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.603 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6826", 2023-11-01T04:16:12.603 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3465541770 2023-11-01T04:16:12.603 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.603 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.603 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.603 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6827", 2023-11-01T04:16:12.604 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3465541770 2023-11-01T04:16:12.604 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.604 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.604 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addrs": { 2023-11-01T04:16:12.604 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.605 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.605 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.605 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6830", 2023-11-01T04:16:12.605 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3465541770 2023-11-01T04:16:12.605 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.606 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.606 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.606 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6831", 2023-11-01T04:16:12.606 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3465541770 2023-11-01T04:16:12.607 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.607 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.607 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addrs": { 2023-11-01T04:16:12.607 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.607 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.608 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.608 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6828", 2023-11-01T04:16:12.608 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3465541770 2023-11-01T04:16:12.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.608 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.609 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.609 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6829", 2023-11-01T04:16:12.609 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3465541770 2023-11-01T04:16:12.609 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.609 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.610 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6822/3465541770", 2023-11-01T04:16:12.610 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "172.21.15.181:6827/3465541770", 2023-11-01T04:16:12.610 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addr": "172.21.15.181:6831/3465541770", 2023-11-01T04:16:12.610 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addr": "172.21.15.181:6829/3465541770", 2023-11-01T04:16:12.610 INFO:teuthology.orchestra.run.smithi153.stdout: "state": [ 2023-11-01T04:16:12.611 INFO:teuthology.orchestra.run.smithi153.stdout: "exists", 2023-11-01T04:16:12.611 INFO:teuthology.orchestra.run.smithi153.stdout: "up" 2023-11-01T04:16:12.611 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.611 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.611 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 7, 2023-11-01T04:16:12.612 INFO:teuthology.orchestra.run.smithi153.stdout: "uuid": "1e80e965-2a46-468c-bcb8-49bee8dd19c1", 2023-11-01T04:16:12.612 INFO:teuthology.orchestra.run.smithi153.stdout: "up": 1, 2023-11-01T04:16:12.612 INFO:teuthology.orchestra.run.smithi153.stdout: "in": 1, 2023-11-01T04:16:12.612 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 1, 2023-11-01T04:16:12.612 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity": 1, 2023-11-01T04:16:12.612 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_begin": 0, 2023-11-01T04:16:12.613 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_end": 0, 2023-11-01T04:16:12.613 INFO:teuthology.orchestra.run.smithi153.stdout: "up_from": 22, 2023-11-01T04:16:12.613 INFO:teuthology.orchestra.run.smithi153.stdout: "up_thru": 31, 2023-11-01T04:16:12.613 INFO:teuthology.orchestra.run.smithi153.stdout: "down_at": 0, 2023-11-01T04:16:12.613 INFO:teuthology.orchestra.run.smithi153.stdout: "lost_at": 0, 2023-11-01T04:16:12.614 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-11-01T04:16:12.614 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.614 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.614 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.614 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6802", 2023-11-01T04:16:12.614 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1763196448 2023-11-01T04:16:12.615 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.615 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.615 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.615 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6803", 2023-11-01T04:16:12.615 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1763196448 2023-11-01T04:16:12.616 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.616 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.616 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addrs": { 2023-11-01T04:16:12.616 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.616 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.617 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.617 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6804", 2023-11-01T04:16:12.617 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1763196448 2023-11-01T04:16:12.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.617 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.617 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.617 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6805", 2023-11-01T04:16:12.618 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1763196448 2023-11-01T04:16:12.618 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.618 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.618 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.618 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addrs": { 2023-11-01T04:16:12.618 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.619 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.619 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.619 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6808", 2023-11-01T04:16:12.619 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1763196448 2023-11-01T04:16:12.619 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.619 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.620 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.620 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6809", 2023-11-01T04:16:12.620 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1763196448 2023-11-01T04:16:12.620 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.620 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.621 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addrs": { 2023-11-01T04:16:12.621 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.621 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.621 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.621 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6806", 2023-11-01T04:16:12.622 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1763196448 2023-11-01T04:16:12.622 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.622 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.622 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.622 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6807", 2023-11-01T04:16:12.622 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1763196448 2023-11-01T04:16:12.623 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.623 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.623 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.190:6803/1763196448", 2023-11-01T04:16:12.623 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "172.21.15.190:6805/1763196448", 2023-11-01T04:16:12.623 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addr": "172.21.15.190:6809/1763196448", 2023-11-01T04:16:12.624 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addr": "172.21.15.190:6807/1763196448", 2023-11-01T04:16:12.624 INFO:teuthology.orchestra.run.smithi153.stdout: "state": [ 2023-11-01T04:16:12.624 INFO:teuthology.orchestra.run.smithi153.stdout: "exists", 2023-11-01T04:16:12.624 INFO:teuthology.orchestra.run.smithi153.stdout: "up" 2023-11-01T04:16:12.624 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.625 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.625 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 8, 2023-11-01T04:16:12.625 INFO:teuthology.orchestra.run.smithi153.stdout: "uuid": "87179b74-818f-47df-ad70-1392cfe227e9", 2023-11-01T04:16:12.625 INFO:teuthology.orchestra.run.smithi153.stdout: "up": 1, 2023-11-01T04:16:12.625 INFO:teuthology.orchestra.run.smithi153.stdout: "in": 1, 2023-11-01T04:16:12.625 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 1, 2023-11-01T04:16:12.626 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity": 1, 2023-11-01T04:16:12.626 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_begin": 0, 2023-11-01T04:16:12.626 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_end": 0, 2023-11-01T04:16:12.626 INFO:teuthology.orchestra.run.smithi153.stdout: "up_from": 22, 2023-11-01T04:16:12.626 INFO:teuthology.orchestra.run.smithi153.stdout: "up_thru": 31, 2023-11-01T04:16:12.626 INFO:teuthology.orchestra.run.smithi153.stdout: "down_at": 0, 2023-11-01T04:16:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: "lost_at": 0, 2023-11-01T04:16:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-11-01T04:16:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.627 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6800", 2023-11-01T04:16:12.628 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1068520389 2023-11-01T04:16:12.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.628 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.628 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.628 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6801", 2023-11-01T04:16:12.628 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1068520389 2023-11-01T04:16:12.629 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.629 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.629 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addrs": { 2023-11-01T04:16:12.629 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.629 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.630 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.630 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6802", 2023-11-01T04:16:12.630 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1068520389 2023-11-01T04:16:12.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.630 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.631 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.631 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6803", 2023-11-01T04:16:12.631 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1068520389 2023-11-01T04:16:12.631 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.631 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.632 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addrs": { 2023-11-01T04:16:12.632 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.632 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.632 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.632 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6806", 2023-11-01T04:16:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1068520389 2023-11-01T04:16:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6807", 2023-11-01T04:16:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1068520389 2023-11-01T04:16:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addrs": { 2023-11-01T04:16:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6804", 2023-11-01T04:16:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1068520389 2023-11-01T04:16:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6805", 2023-11-01T04:16:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 1068520389 2023-11-01T04:16:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6801/1068520389", 2023-11-01T04:16:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "172.21.15.153:6803/1068520389", 2023-11-01T04:16:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addr": "172.21.15.153:6807/1068520389", 2023-11-01T04:16:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addr": "172.21.15.153:6805/1068520389", 2023-11-01T04:16:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: "state": [ 2023-11-01T04:16:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: "exists", 2023-11-01T04:16:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: "up" 2023-11-01T04:16:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 9, 2023-11-01T04:16:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: "uuid": "10d3c443-1748-4a7c-8b60-b58cfd565b4d", 2023-11-01T04:16:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: "up": 1, 2023-11-01T04:16:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: "in": 1, 2023-11-01T04:16:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 1, 2023-11-01T04:16:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity": 1, 2023-11-01T04:16:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_begin": 0, 2023-11-01T04:16:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_end": 0, 2023-11-01T04:16:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: "up_from": 22, 2023-11-01T04:16:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: "up_thru": 0, 2023-11-01T04:16:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: "down_at": 0, 2023-11-01T04:16:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: "lost_at": 0, 2023-11-01T04:16:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-11-01T04:16:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6800", 2023-11-01T04:16:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 210565898 2023-11-01T04:16:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6801", 2023-11-01T04:16:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 210565898 2023-11-01T04:16:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addrs": { 2023-11-01T04:16:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6802", 2023-11-01T04:16:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 210565898 2023-11-01T04:16:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6803", 2023-11-01T04:16:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 210565898 2023-11-01T04:16:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addrs": { 2023-11-01T04:16:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6806", 2023-11-01T04:16:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 210565898 2023-11-01T04:16:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6807", 2023-11-01T04:16:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 210565898 2023-11-01T04:16:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addrs": { 2023-11-01T04:16:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6804", 2023-11-01T04:16:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 210565898 2023-11-01T04:16:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6805", 2023-11-01T04:16:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 210565898 2023-11-01T04:16:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.173:6801/210565898", 2023-11-01T04:16:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "172.21.15.173:6803/210565898", 2023-11-01T04:16:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addr": "172.21.15.173:6807/210565898", 2023-11-01T04:16:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addr": "172.21.15.173:6805/210565898", 2023-11-01T04:16:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: "state": [ 2023-11-01T04:16:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: "exists", 2023-11-01T04:16:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: "up" 2023-11-01T04:16:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 10, 2023-11-01T04:16:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: "uuid": "81bb6569-d02b-4638-8b76-f99756d415b5", 2023-11-01T04:16:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: "up": 1, 2023-11-01T04:16:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: "in": 1, 2023-11-01T04:16:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 1, 2023-11-01T04:16:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity": 1, 2023-11-01T04:16:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_begin": 0, 2023-11-01T04:16:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_end": 0, 2023-11-01T04:16:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: "up_from": 22, 2023-11-01T04:16:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: "up_thru": 31, 2023-11-01T04:16:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: "down_at": 0, 2023-11-01T04:16:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: "lost_at": 0, 2023-11-01T04:16:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-11-01T04:16:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6800", 2023-11-01T04:16:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2007605188 2023-11-01T04:16:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6801", 2023-11-01T04:16:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2007605188 2023-11-01T04:16:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addrs": { 2023-11-01T04:16:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6802", 2023-11-01T04:16:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2007605188 2023-11-01T04:16:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6803", 2023-11-01T04:16:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2007605188 2023-11-01T04:16:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addrs": { 2023-11-01T04:16:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6806", 2023-11-01T04:16:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2007605188 2023-11-01T04:16:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6807", 2023-11-01T04:16:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2007605188 2023-11-01T04:16:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addrs": { 2023-11-01T04:16:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6804", 2023-11-01T04:16:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2007605188 2023-11-01T04:16:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6805", 2023-11-01T04:16:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2007605188 2023-11-01T04:16:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6801/2007605188", 2023-11-01T04:16:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "172.21.15.181:6803/2007605188", 2023-11-01T04:16:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addr": "172.21.15.181:6807/2007605188", 2023-11-01T04:16:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addr": "172.21.15.181:6805/2007605188", 2023-11-01T04:16:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: "state": [ 2023-11-01T04:16:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: "exists", 2023-11-01T04:16:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: "up" 2023-11-01T04:16:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 11, 2023-11-01T04:16:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: "uuid": "b8aefe11-845f-48a4-930c-b05cdfbb460a", 2023-11-01T04:16:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: "up": 1, 2023-11-01T04:16:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: "in": 1, 2023-11-01T04:16:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 1, 2023-11-01T04:16:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity": 1, 2023-11-01T04:16:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_begin": 0, 2023-11-01T04:16:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_end": 0, 2023-11-01T04:16:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: "up_from": 22, 2023-11-01T04:16:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: "up_thru": 31, 2023-11-01T04:16:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: "down_at": 0, 2023-11-01T04:16:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: "lost_at": 0, 2023-11-01T04:16:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-11-01T04:16:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6810", 2023-11-01T04:16:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2456507663 2023-11-01T04:16:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6812", 2023-11-01T04:16:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2456507663 2023-11-01T04:16:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addrs": { 2023-11-01T04:16:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6814", 2023-11-01T04:16:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2456507663 2023-11-01T04:16:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6816", 2023-11-01T04:16:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2456507663 2023-11-01T04:16:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addrs": { 2023-11-01T04:16:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6822", 2023-11-01T04:16:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2456507663 2023-11-01T04:16:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6824", 2023-11-01T04:16:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2456507663 2023-11-01T04:16:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addrs": { 2023-11-01T04:16:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6818", 2023-11-01T04:16:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2456507663 2023-11-01T04:16:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6820", 2023-11-01T04:16:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2456507663 2023-11-01T04:16:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.190:6812/2456507663", 2023-11-01T04:16:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "172.21.15.190:6816/2456507663", 2023-11-01T04:16:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addr": "172.21.15.190:6824/2456507663", 2023-11-01T04:16:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addr": "172.21.15.190:6820/2456507663", 2023-11-01T04:16:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: "state": [ 2023-11-01T04:16:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: "exists", 2023-11-01T04:16:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: "up" 2023-11-01T04:16:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 12, 2023-11-01T04:16:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: "uuid": "2165437d-26ba-4e50-8646-8cedbc10da53", 2023-11-01T04:16:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: "up": 1, 2023-11-01T04:16:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: "in": 1, 2023-11-01T04:16:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 1, 2023-11-01T04:16:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity": 1, 2023-11-01T04:16:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_begin": 0, 2023-11-01T04:16:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_end": 0, 2023-11-01T04:16:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: "up_from": 22, 2023-11-01T04:16:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "up_thru": 31, 2023-11-01T04:16:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "down_at": 0, 2023-11-01T04:16:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "lost_at": 0, 2023-11-01T04:16:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-11-01T04:16:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6808", 2023-11-01T04:16:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 819245857 2023-11-01T04:16:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6809", 2023-11-01T04:16:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 819245857 2023-11-01T04:16:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addrs": { 2023-11-01T04:16:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6810", 2023-11-01T04:16:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 819245857 2023-11-01T04:16:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6811", 2023-11-01T04:16:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 819245857 2023-11-01T04:16:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addrs": { 2023-11-01T04:16:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6814", 2023-11-01T04:16:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 819245857 2023-11-01T04:16:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6815", 2023-11-01T04:16:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 819245857 2023-11-01T04:16:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addrs": { 2023-11-01T04:16:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6812", 2023-11-01T04:16:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 819245857 2023-11-01T04:16:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6813", 2023-11-01T04:16:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 819245857 2023-11-01T04:16:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6809/819245857", 2023-11-01T04:16:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "172.21.15.153:6811/819245857", 2023-11-01T04:16:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addr": "172.21.15.153:6815/819245857", 2023-11-01T04:16:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addr": "172.21.15.153:6813/819245857", 2023-11-01T04:16:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: "state": [ 2023-11-01T04:16:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: "exists", 2023-11-01T04:16:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: "up" 2023-11-01T04:16:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 13, 2023-11-01T04:16:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "uuid": "7b9f38b4-f5ff-43f1-a373-2aaed710026d", 2023-11-01T04:16:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "up": 1, 2023-11-01T04:16:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "in": 1, 2023-11-01T04:16:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 1, 2023-11-01T04:16:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity": 1, 2023-11-01T04:16:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_begin": 0, 2023-11-01T04:16:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_end": 0, 2023-11-01T04:16:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: "up_from": 22, 2023-11-01T04:16:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: "up_thru": 31, 2023-11-01T04:16:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: "down_at": 0, 2023-11-01T04:16:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: "lost_at": 0, 2023-11-01T04:16:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-11-01T04:16:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6808", 2023-11-01T04:16:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3482699195 2023-11-01T04:16:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6809", 2023-11-01T04:16:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3482699195 2023-11-01T04:16:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addrs": { 2023-11-01T04:16:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6810", 2023-11-01T04:16:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3482699195 2023-11-01T04:16:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6811", 2023-11-01T04:16:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3482699195 2023-11-01T04:16:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addrs": { 2023-11-01T04:16:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6816", 2023-11-01T04:16:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3482699195 2023-11-01T04:16:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6819", 2023-11-01T04:16:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3482699195 2023-11-01T04:16:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addrs": { 2023-11-01T04:16:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6812", 2023-11-01T04:16:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3482699195 2023-11-01T04:16:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.173:6814", 2023-11-01T04:16:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3482699195 2023-11-01T04:16:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.173:6809/3482699195", 2023-11-01T04:16:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "172.21.15.173:6811/3482699195", 2023-11-01T04:16:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addr": "172.21.15.173:6819/3482699195", 2023-11-01T04:16:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addr": "172.21.15.173:6814/3482699195", 2023-11-01T04:16:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: "state": [ 2023-11-01T04:16:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: "exists", 2023-11-01T04:16:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: "up" 2023-11-01T04:16:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 14, 2023-11-01T04:16:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: "uuid": "054b27d2-8a4d-4851-8af8-78b71e6a537e", 2023-11-01T04:16:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: "up": 1, 2023-11-01T04:16:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: "in": 1, 2023-11-01T04:16:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 1, 2023-11-01T04:16:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity": 1, 2023-11-01T04:16:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_begin": 0, 2023-11-01T04:16:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_end": 0, 2023-11-01T04:16:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: "up_from": 22, 2023-11-01T04:16:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: "up_thru": 31, 2023-11-01T04:16:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: "down_at": 0, 2023-11-01T04:16:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: "lost_at": 0, 2023-11-01T04:16:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-11-01T04:16:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6808", 2023-11-01T04:16:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2780345165 2023-11-01T04:16:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6809", 2023-11-01T04:16:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2780345165 2023-11-01T04:16:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addrs": { 2023-11-01T04:16:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6810", 2023-11-01T04:16:12.714 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2780345165 2023-11-01T04:16:12.714 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.714 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.714 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.714 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6811", 2023-11-01T04:16:12.714 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2780345165 2023-11-01T04:16:12.715 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.715 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.715 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addrs": { 2023-11-01T04:16:12.715 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.715 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.716 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.716 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6814", 2023-11-01T04:16:12.716 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2780345165 2023-11-01T04:16:12.716 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.716 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.716 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.717 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6815", 2023-11-01T04:16:12.717 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2780345165 2023-11-01T04:16:12.717 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.717 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.717 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.717 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addrs": { 2023-11-01T04:16:12.718 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.718 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.718 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.718 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6812", 2023-11-01T04:16:12.718 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2780345165 2023-11-01T04:16:12.718 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.719 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.719 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.719 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.181:6813", 2023-11-01T04:16:12.719 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 2780345165 2023-11-01T04:16:12.719 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.719 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.720 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.720 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.181:6809/2780345165", 2023-11-01T04:16:12.720 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "172.21.15.181:6811/2780345165", 2023-11-01T04:16:12.720 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addr": "172.21.15.181:6815/2780345165", 2023-11-01T04:16:12.720 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addr": "172.21.15.181:6813/2780345165", 2023-11-01T04:16:12.720 INFO:teuthology.orchestra.run.smithi153.stdout: "state": [ 2023-11-01T04:16:12.721 INFO:teuthology.orchestra.run.smithi153.stdout: "exists", 2023-11-01T04:16:12.721 INFO:teuthology.orchestra.run.smithi153.stdout: "up" 2023-11-01T04:16:12.721 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.721 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.721 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.721 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 15, 2023-11-01T04:16:12.722 INFO:teuthology.orchestra.run.smithi153.stdout: "uuid": "e650f936-3712-47cd-9b17-d732991a27a3", 2023-11-01T04:16:12.722 INFO:teuthology.orchestra.run.smithi153.stdout: "up": 1, 2023-11-01T04:16:12.722 INFO:teuthology.orchestra.run.smithi153.stdout: "in": 1, 2023-11-01T04:16:12.722 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 1, 2023-11-01T04:16:12.722 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_affinity": 1, 2023-11-01T04:16:12.722 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_begin": 0, 2023-11-01T04:16:12.723 INFO:teuthology.orchestra.run.smithi153.stdout: "last_clean_end": 0, 2023-11-01T04:16:12.723 INFO:teuthology.orchestra.run.smithi153.stdout: "up_from": 22, 2023-11-01T04:16:12.723 INFO:teuthology.orchestra.run.smithi153.stdout: "up_thru": 0, 2023-11-01T04:16:12.723 INFO:teuthology.orchestra.run.smithi153.stdout: "down_at": 0, 2023-11-01T04:16:12.723 INFO:teuthology.orchestra.run.smithi153.stdout: "lost_at": 0, 2023-11-01T04:16:12.724 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2023-11-01T04:16:12.724 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.724 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.724 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.724 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6811", 2023-11-01T04:16:12.724 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3871145029 2023-11-01T04:16:12.725 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.725 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.725 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.725 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6813", 2023-11-01T04:16:12.725 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3871145029 2023-11-01T04:16:12.725 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.725 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.726 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.726 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addrs": { 2023-11-01T04:16:12.726 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.726 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.726 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.727 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6815", 2023-11-01T04:16:12.727 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3871145029 2023-11-01T04:16:12.727 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.727 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.727 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.727 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6817", 2023-11-01T04:16:12.728 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3871145029 2023-11-01T04:16:12.728 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.728 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.728 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.728 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addrs": { 2023-11-01T04:16:12.728 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.729 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.729 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.729 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6823", 2023-11-01T04:16:12.729 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3871145029 2023-11-01T04:16:12.729 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.729 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.730 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.730 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6825", 2023-11-01T04:16:12.730 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3871145029 2023-11-01T04:16:12.730 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.730 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.730 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.731 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addrs": { 2023-11-01T04:16:12.731 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2023-11-01T04:16:12.731 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.731 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2023-11-01T04:16:12.731 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6819", 2023-11-01T04:16:12.731 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3871145029 2023-11-01T04:16:12.732 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.732 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.732 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2023-11-01T04:16:12.732 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.190:6821", 2023-11-01T04:16:12.732 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 3871145029 2023-11-01T04:16:12.732 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.733 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.733 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.733 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.190:6813/3871145029", 2023-11-01T04:16:12.733 INFO:teuthology.orchestra.run.smithi153.stdout: "cluster_addr": "172.21.15.190:6817/3871145029", 2023-11-01T04:16:12.733 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_back_addr": "172.21.15.190:6825/3871145029", 2023-11-01T04:16:12.733 INFO:teuthology.orchestra.run.smithi153.stdout: "heartbeat_front_addr": "172.21.15.190:6821/3871145029", 2023-11-01T04:16:12.734 INFO:teuthology.orchestra.run.smithi153.stdout: "state": [ 2023-11-01T04:16:12.734 INFO:teuthology.orchestra.run.smithi153.stdout: "exists", 2023-11-01T04:16:12.734 INFO:teuthology.orchestra.run.smithi153.stdout: "up" 2023-11-01T04:16:12.734 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:12.734 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.735 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-11-01T04:16:12.735 INFO:teuthology.orchestra.run.smithi153.stdout: "osd_xinfo": [ 2023-11-01T04:16:12.735 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.735 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 0, 2023-11-01T04:16:12.735 INFO:teuthology.orchestra.run.smithi153.stdout: "down_stamp": "0.000000", 2023-11-01T04:16:12.735 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_probability": 0, 2023-11-01T04:16:12.736 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_interval": 0, 2023-11-01T04:16:12.736 INFO:teuthology.orchestra.run.smithi153.stdout: "features": 4540138322906710015, 2023-11-01T04:16:12.736 INFO:teuthology.orchestra.run.smithi153.stdout: "old_weight": 0, 2023-11-01T04:16:12.736 INFO:teuthology.orchestra.run.smithi153.stdout: "last_purged_snaps_scrub": "2023-11-01T04:15:54.898068+0000", 2023-11-01T04:16:12.736 INFO:teuthology.orchestra.run.smithi153.stdout: "dead_epoch": 0 2023-11-01T04:16:12.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.736 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.737 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 1, 2023-11-01T04:16:12.737 INFO:teuthology.orchestra.run.smithi153.stdout: "down_stamp": "0.000000", 2023-11-01T04:16:12.737 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_probability": 0, 2023-11-01T04:16:12.737 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_interval": 0, 2023-11-01T04:16:12.737 INFO:teuthology.orchestra.run.smithi153.stdout: "features": 4540138322906710015, 2023-11-01T04:16:12.737 INFO:teuthology.orchestra.run.smithi153.stdout: "old_weight": 0, 2023-11-01T04:16:12.738 INFO:teuthology.orchestra.run.smithi153.stdout: "last_purged_snaps_scrub": "2023-11-01T04:15:54.918555+0000", 2023-11-01T04:16:12.738 INFO:teuthology.orchestra.run.smithi153.stdout: "dead_epoch": 0 2023-11-01T04:16:12.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.738 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.738 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 2, 2023-11-01T04:16:12.738 INFO:teuthology.orchestra.run.smithi153.stdout: "down_stamp": "0.000000", 2023-11-01T04:16:12.739 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_probability": 0, 2023-11-01T04:16:12.739 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_interval": 0, 2023-11-01T04:16:12.739 INFO:teuthology.orchestra.run.smithi153.stdout: "features": 4540138322906710015, 2023-11-01T04:16:12.739 INFO:teuthology.orchestra.run.smithi153.stdout: "old_weight": 0, 2023-11-01T04:16:12.739 INFO:teuthology.orchestra.run.smithi153.stdout: "last_purged_snaps_scrub": "2023-11-01T04:15:55.123290+0000", 2023-11-01T04:16:12.739 INFO:teuthology.orchestra.run.smithi153.stdout: "dead_epoch": 0 2023-11-01T04:16:12.740 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.740 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.740 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 3, 2023-11-01T04:16:12.740 INFO:teuthology.orchestra.run.smithi153.stdout: "down_stamp": "0.000000", 2023-11-01T04:16:12.740 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_probability": 0, 2023-11-01T04:16:12.740 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_interval": 0, 2023-11-01T04:16:12.741 INFO:teuthology.orchestra.run.smithi153.stdout: "features": 4540138322906710015, 2023-11-01T04:16:12.741 INFO:teuthology.orchestra.run.smithi153.stdout: "old_weight": 0, 2023-11-01T04:16:12.741 INFO:teuthology.orchestra.run.smithi153.stdout: "last_purged_snaps_scrub": "2023-11-01T04:15:54.920765+0000", 2023-11-01T04:16:12.741 INFO:teuthology.orchestra.run.smithi153.stdout: "dead_epoch": 0 2023-11-01T04:16:12.741 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.742 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.742 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 4, 2023-11-01T04:16:12.742 INFO:teuthology.orchestra.run.smithi153.stdout: "down_stamp": "0.000000", 2023-11-01T04:16:12.742 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_probability": 0, 2023-11-01T04:16:12.742 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_interval": 0, 2023-11-01T04:16:12.742 INFO:teuthology.orchestra.run.smithi153.stdout: "features": 4540138322906710015, 2023-11-01T04:16:12.743 INFO:teuthology.orchestra.run.smithi153.stdout: "old_weight": 0, 2023-11-01T04:16:12.743 INFO:teuthology.orchestra.run.smithi153.stdout: "last_purged_snaps_scrub": "2023-11-01T04:15:54.941559+0000", 2023-11-01T04:16:12.743 INFO:teuthology.orchestra.run.smithi153.stdout: "dead_epoch": 0 2023-11-01T04:16:12.743 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.743 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.743 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 5, 2023-11-01T04:16:12.744 INFO:teuthology.orchestra.run.smithi153.stdout: "down_stamp": "0.000000", 2023-11-01T04:16:12.744 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_probability": 0, 2023-11-01T04:16:12.744 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_interval": 0, 2023-11-01T04:16:12.744 INFO:teuthology.orchestra.run.smithi153.stdout: "features": 4540138322906710015, 2023-11-01T04:16:12.744 INFO:teuthology.orchestra.run.smithi153.stdout: "old_weight": 0, 2023-11-01T04:16:12.744 INFO:teuthology.orchestra.run.smithi153.stdout: "last_purged_snaps_scrub": "2023-11-01T04:15:55.003678+0000", 2023-11-01T04:16:12.745 INFO:teuthology.orchestra.run.smithi153.stdout: "dead_epoch": 0 2023-11-01T04:16:12.745 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.745 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.745 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 6, 2023-11-01T04:16:12.745 INFO:teuthology.orchestra.run.smithi153.stdout: "down_stamp": "0.000000", 2023-11-01T04:16:12.745 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_probability": 0, 2023-11-01T04:16:12.746 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_interval": 0, 2023-11-01T04:16:12.746 INFO:teuthology.orchestra.run.smithi153.stdout: "features": 4540138322906710015, 2023-11-01T04:16:12.746 INFO:teuthology.orchestra.run.smithi153.stdout: "old_weight": 0, 2023-11-01T04:16:12.746 INFO:teuthology.orchestra.run.smithi153.stdout: "last_purged_snaps_scrub": "2023-11-01T04:15:54.002527+0000", 2023-11-01T04:16:12.746 INFO:teuthology.orchestra.run.smithi153.stdout: "dead_epoch": 0 2023-11-01T04:16:12.746 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.747 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.747 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 7, 2023-11-01T04:16:12.747 INFO:teuthology.orchestra.run.smithi153.stdout: "down_stamp": "0.000000", 2023-11-01T04:16:12.747 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_probability": 0, 2023-11-01T04:16:12.747 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_interval": 0, 2023-11-01T04:16:12.748 INFO:teuthology.orchestra.run.smithi153.stdout: "features": 4540138322906710015, 2023-11-01T04:16:12.748 INFO:teuthology.orchestra.run.smithi153.stdout: "old_weight": 0, 2023-11-01T04:16:12.748 INFO:teuthology.orchestra.run.smithi153.stdout: "last_purged_snaps_scrub": "2023-11-01T04:15:55.028107+0000", 2023-11-01T04:16:12.748 INFO:teuthology.orchestra.run.smithi153.stdout: "dead_epoch": 0 2023-11-01T04:16:12.748 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.748 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.749 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 8, 2023-11-01T04:16:12.749 INFO:teuthology.orchestra.run.smithi153.stdout: "down_stamp": "0.000000", 2023-11-01T04:16:12.749 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_probability": 0, 2023-11-01T04:16:12.749 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_interval": 0, 2023-11-01T04:16:12.749 INFO:teuthology.orchestra.run.smithi153.stdout: "features": 4540138322906710015, 2023-11-01T04:16:12.749 INFO:teuthology.orchestra.run.smithi153.stdout: "old_weight": 0, 2023-11-01T04:16:12.750 INFO:teuthology.orchestra.run.smithi153.stdout: "last_purged_snaps_scrub": "2023-11-01T04:15:54.920344+0000", 2023-11-01T04:16:12.750 INFO:teuthology.orchestra.run.smithi153.stdout: "dead_epoch": 0 2023-11-01T04:16:12.750 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.750 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.750 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 9, 2023-11-01T04:16:12.751 INFO:teuthology.orchestra.run.smithi153.stdout: "down_stamp": "0.000000", 2023-11-01T04:16:12.751 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_probability": 0, 2023-11-01T04:16:12.751 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_interval": 0, 2023-11-01T04:16:12.751 INFO:teuthology.orchestra.run.smithi153.stdout: "features": 4540138322906710015, 2023-11-01T04:16:12.751 INFO:teuthology.orchestra.run.smithi153.stdout: "old_weight": 0, 2023-11-01T04:16:12.752 INFO:teuthology.orchestra.run.smithi153.stdout: "last_purged_snaps_scrub": "2023-11-01T04:15:54.893447+0000", 2023-11-01T04:16:12.752 INFO:teuthology.orchestra.run.smithi153.stdout: "dead_epoch": 0 2023-11-01T04:16:12.752 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.752 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.752 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 10, 2023-11-01T04:16:12.752 INFO:teuthology.orchestra.run.smithi153.stdout: "down_stamp": "0.000000", 2023-11-01T04:16:12.753 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_probability": 0, 2023-11-01T04:16:12.753 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_interval": 0, 2023-11-01T04:16:12.753 INFO:teuthology.orchestra.run.smithi153.stdout: "features": 4540138322906710015, 2023-11-01T04:16:12.753 INFO:teuthology.orchestra.run.smithi153.stdout: "old_weight": 0, 2023-11-01T04:16:12.753 INFO:teuthology.orchestra.run.smithi153.stdout: "last_purged_snaps_scrub": "2023-11-01T04:15:55.013907+0000", 2023-11-01T04:16:12.753 INFO:teuthology.orchestra.run.smithi153.stdout: "dead_epoch": 0 2023-11-01T04:16:12.754 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.754 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.754 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 11, 2023-11-01T04:16:12.754 INFO:teuthology.orchestra.run.smithi153.stdout: "down_stamp": "0.000000", 2023-11-01T04:16:12.754 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_probability": 0, 2023-11-01T04:16:12.755 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_interval": 0, 2023-11-01T04:16:12.755 INFO:teuthology.orchestra.run.smithi153.stdout: "features": 4540138322906710015, 2023-11-01T04:16:12.755 INFO:teuthology.orchestra.run.smithi153.stdout: "old_weight": 0, 2023-11-01T04:16:12.755 INFO:teuthology.orchestra.run.smithi153.stdout: "last_purged_snaps_scrub": "2023-11-01T04:15:54.898025+0000", 2023-11-01T04:16:12.755 INFO:teuthology.orchestra.run.smithi153.stdout: "dead_epoch": 0 2023-11-01T04:16:12.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.756 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.756 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 12, 2023-11-01T04:16:12.756 INFO:teuthology.orchestra.run.smithi153.stdout: "down_stamp": "0.000000", 2023-11-01T04:16:12.756 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_probability": 0, 2023-11-01T04:16:12.756 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_interval": 0, 2023-11-01T04:16:12.757 INFO:teuthology.orchestra.run.smithi153.stdout: "features": 4540138322906710015, 2023-11-01T04:16:12.757 INFO:teuthology.orchestra.run.smithi153.stdout: "old_weight": 0, 2023-11-01T04:16:12.757 INFO:teuthology.orchestra.run.smithi153.stdout: "last_purged_snaps_scrub": "2023-11-01T04:15:54.650893+0000", 2023-11-01T04:16:12.757 INFO:teuthology.orchestra.run.smithi153.stdout: "dead_epoch": 0 2023-11-01T04:16:12.757 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.757 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.758 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 13, 2023-11-01T04:16:12.758 INFO:teuthology.orchestra.run.smithi153.stdout: "down_stamp": "0.000000", 2023-11-01T04:16:12.758 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_probability": 0, 2023-11-01T04:16:12.758 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_interval": 0, 2023-11-01T04:16:12.758 INFO:teuthology.orchestra.run.smithi153.stdout: "features": 4540138322906710015, 2023-11-01T04:16:12.759 INFO:teuthology.orchestra.run.smithi153.stdout: "old_weight": 0, 2023-11-01T04:16:12.759 INFO:teuthology.orchestra.run.smithi153.stdout: "last_purged_snaps_scrub": "2023-11-01T04:15:55.003356+0000", 2023-11-01T04:16:12.759 INFO:teuthology.orchestra.run.smithi153.stdout: "dead_epoch": 0 2023-11-01T04:16:12.759 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.759 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.759 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 14, 2023-11-01T04:16:12.760 INFO:teuthology.orchestra.run.smithi153.stdout: "down_stamp": "0.000000", 2023-11-01T04:16:12.760 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_probability": 0, 2023-11-01T04:16:12.760 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_interval": 0, 2023-11-01T04:16:12.760 INFO:teuthology.orchestra.run.smithi153.stdout: "features": 4540138322906710015, 2023-11-01T04:16:12.760 INFO:teuthology.orchestra.run.smithi153.stdout: "old_weight": 0, 2023-11-01T04:16:12.760 INFO:teuthology.orchestra.run.smithi153.stdout: "last_purged_snaps_scrub": "2023-11-01T04:15:55.008057+0000", 2023-11-01T04:16:12.761 INFO:teuthology.orchestra.run.smithi153.stdout: "dead_epoch": 0 2023-11-01T04:16:12.761 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.761 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:12.761 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": 15, 2023-11-01T04:16:12.761 INFO:teuthology.orchestra.run.smithi153.stdout: "down_stamp": "0.000000", 2023-11-01T04:16:12.761 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_probability": 0, 2023-11-01T04:16:12.762 INFO:teuthology.orchestra.run.smithi153.stdout: "laggy_interval": 0, 2023-11-01T04:16:12.762 INFO:teuthology.orchestra.run.smithi153.stdout: "features": 4540138322906710015, 2023-11-01T04:16:12.762 INFO:teuthology.orchestra.run.smithi153.stdout: "old_weight": 0, 2023-11-01T04:16:12.762 INFO:teuthology.orchestra.run.smithi153.stdout: "last_purged_snaps_scrub": "2023-11-01T04:15:55.065472+0000", 2023-11-01T04:16:12.762 INFO:teuthology.orchestra.run.smithi153.stdout: "dead_epoch": 0 2023-11-01T04:16:12.762 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.763 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-11-01T04:16:12.763 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_upmap": [], 2023-11-01T04:16:12.763 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_upmap_items": [], 2023-11-01T04:16:12.763 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_upmap_primaries": [], 2023-11-01T04:16:12.763 INFO:teuthology.orchestra.run.smithi153.stdout: "pg_temp": [], 2023-11-01T04:16:12.763 INFO:teuthology.orchestra.run.smithi153.stdout: "primary_temp": [], 2023-11-01T04:16:12.764 INFO:teuthology.orchestra.run.smithi153.stdout: "blocklist": {}, 2023-11-01T04:16:12.764 INFO:teuthology.orchestra.run.smithi153.stdout: "range_blocklist": {}, 2023-11-01T04:16:12.764 INFO:teuthology.orchestra.run.smithi153.stdout: "erasure_code_profiles": { 2023-11-01T04:16:12.764 INFO:teuthology.orchestra.run.smithi153.stdout: "default": { 2023-11-01T04:16:12.764 INFO:teuthology.orchestra.run.smithi153.stdout: "crush-failure-domain": "osd", 2023-11-01T04:16:12.764 INFO:teuthology.orchestra.run.smithi153.stdout: "k": "2", 2023-11-01T04:16:12.765 INFO:teuthology.orchestra.run.smithi153.stdout: "m": "1", 2023-11-01T04:16:12.765 INFO:teuthology.orchestra.run.smithi153.stdout: "plugin": "jerasure", 2023-11-01T04:16:12.765 INFO:teuthology.orchestra.run.smithi153.stdout: "technique": "reed_sol_van" 2023-11-01T04:16:12.765 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.765 INFO:teuthology.orchestra.run.smithi153.stdout: "shecprofile": { 2023-11-01T04:16:12.766 INFO:teuthology.orchestra.run.smithi153.stdout: "c": "2", 2023-11-01T04:16:12.766 INFO:teuthology.orchestra.run.smithi153.stdout: "crush-device-class": "", 2023-11-01T04:16:12.766 INFO:teuthology.orchestra.run.smithi153.stdout: "crush-failure-domain": "osd", 2023-11-01T04:16:12.766 INFO:teuthology.orchestra.run.smithi153.stdout: "crush-root": "default", 2023-11-01T04:16:12.766 INFO:teuthology.orchestra.run.smithi153.stdout: "k": "4", 2023-11-01T04:16:12.766 INFO:teuthology.orchestra.run.smithi153.stdout: "m": "3", 2023-11-01T04:16:12.767 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "shecprofile", 2023-11-01T04:16:12.767 INFO:teuthology.orchestra.run.smithi153.stdout: "plugin": "shec", 2023-11-01T04:16:12.767 INFO:teuthology.orchestra.run.smithi153.stdout: "technique": "multiple" 2023-11-01T04:16:12.767 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.767 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:12.767 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_snaps_queue": [], 2023-11-01T04:16:12.767 INFO:teuthology.orchestra.run.smithi153.stdout: "new_removed_snaps": [], 2023-11-01T04:16:12.768 INFO:teuthology.orchestra.run.smithi153.stdout: "new_purged_snaps": [], 2023-11-01T04:16:12.768 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_node_flags": {}, 2023-11-01T04:16:12.768 INFO:teuthology.orchestra.run.smithi153.stdout: "device_class_flags": {}, 2023-11-01T04:16:12.768 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": { 2023-11-01T04:16:12.768 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode_enabled": false, 2023-11-01T04:16:12.769 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_bucket_count": 0, 2023-11-01T04:16:12.769 INFO:teuthology.orchestra.run.smithi153.stdout: "degraded_stretch_mode": 0, 2023-11-01T04:16:12.769 INFO:teuthology.orchestra.run.smithi153.stdout: "recovering_stretch_mode": 0, 2023-11-01T04:16:12.769 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode_bucket": 0 2023-11-01T04:16:12.769 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:12.769 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:12.770 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:12.434+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:12.771 INFO:teuthology.orchestra.run.smithi153.stdout:osd.12: {} 2023-11-01T04:16:12.771 INFO:teuthology.orchestra.run.smithi153.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-01T04:16:12.772 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:12.772 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2023-11-01T04:16:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2023-11-01T04:16:12.773 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [] 2023-11-01T04:16:12.773 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:12.774 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-11-01T04:16:12.775 INFO:tasks.thrashosds.thrasher:j is {'epoch': 33, 'fsid': '0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d', 'created': '2023-11-01T04:15:44.902210+0000', 'modified': '2023-11-01T04:16:11.888692+0000', 'last_up_change': '2023-11-01T04:15:56.634008+0000', 'last_in_change': '2023-11-01T04:15:49.422762+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-01T04:15:57.685107+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-11-01T04:15:59.882949+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 5.989999771118164, 'score_stable': 5.989999771118164, 'optimal_score': 0.6899999976158142, 'raw_score_acting': 4.130000114440918, 'raw_score_stable': 4.130000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2023-11-01T04:16:08.908876+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 7, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'shecprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '83ce3480-defe-4c73-aa1d-47c7941a23ba', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6816', 'nonce': 59702962}, {'type': 'v1', 'addr': '172.21.15.153:6817', 'nonce': 59702962}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6818', 'nonce': 59702962}, {'type': 'v1', 'addr': '172.21.15.153:6819', 'nonce': 59702962}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6822', 'nonce': 59702962}, {'type': 'v1', 'addr': '172.21.15.153:6823', 'nonce': 59702962}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6820', 'nonce': 59702962}, {'type': 'v1', 'addr': '172.21.15.153:6821', 'nonce': 59702962}]}, 'public_addr': '172.21.15.153:6817/59702962', 'cluster_addr': '172.21.15.153:6819/59702962', 'heartbeat_back_addr': '172.21.15.153:6823/59702962', 'heartbeat_front_addr': '172.21.15.153:6821/59702962', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '01a9801f-72fb-4299-8708-d70eb59b0fb2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6813', 'nonce': 3103315277}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 3103315277}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6817', 'nonce': 3103315277}, {'type': 'v1', 'addr': '172.21.15.173:6818', 'nonce': 3103315277}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 3103315277}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 3103315277}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 3103315277}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 3103315277}]}, 'public_addr': '172.21.15.173:6815/3103315277', 'cluster_addr': '172.21.15.173:6818/3103315277', 'heartbeat_back_addr': '172.21.15.173:6823/3103315277', 'heartbeat_front_addr': '172.21.15.173:6821/3103315277', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '7c16db19-9bec-4ca0-88eb-bb5fb673d160', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6816', 'nonce': 1072240131}, {'type': 'v1', 'addr': '172.21.15.181:6817', 'nonce': 1072240131}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6818', 'nonce': 1072240131}, {'type': 'v1', 'addr': '172.21.15.181:6820', 'nonce': 1072240131}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6824', 'nonce': 1072240131}, {'type': 'v1', 'addr': '172.21.15.181:6825', 'nonce': 1072240131}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6821', 'nonce': 1072240131}, {'type': 'v1', 'addr': '172.21.15.181:6823', 'nonce': 1072240131}]}, 'public_addr': '172.21.15.181:6817/1072240131', 'cluster_addr': '172.21.15.181:6820/1072240131', 'heartbeat_back_addr': '172.21.15.181:6825/1072240131', 'heartbeat_front_addr': '172.21.15.181:6823/1072240131', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4c4cc8d1-6761-4d20-90e8-7c3d43d889d8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6826', 'nonce': 2304315422}, {'type': 'v1', 'addr': '172.21.15.190:6827', 'nonce': 2304315422}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6828', 'nonce': 2304315422}, {'type': 'v1', 'addr': '172.21.15.190:6829', 'nonce': 2304315422}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6832', 'nonce': 2304315422}, {'type': 'v1', 'addr': '172.21.15.190:6833', 'nonce': 2304315422}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6830', 'nonce': 2304315422}, {'type': 'v1', 'addr': '172.21.15.190:6831', 'nonce': 2304315422}]}, 'public_addr': '172.21.15.190:6827/2304315422', 'cluster_addr': '172.21.15.190:6829/2304315422', 'heartbeat_back_addr': '172.21.15.190:6833/2304315422', 'heartbeat_front_addr': '172.21.15.190:6831/2304315422', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0fb14630-87e6-4456-9640-7986a5079821', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6824', 'nonce': 3209586931}, {'type': 'v1', 'addr': '172.21.15.153:6825', 'nonce': 3209586931}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6826', 'nonce': 3209586931}, {'type': 'v1', 'addr': '172.21.15.153:6827', 'nonce': 3209586931}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6830', 'nonce': 3209586931}, {'type': 'v1', 'addr': '172.21.15.153:6831', 'nonce': 3209586931}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6828', 'nonce': 3209586931}, {'type': 'v1', 'addr': '172.21.15.153:6829', 'nonce': 3209586931}]}, 'public_addr': '172.21.15.153:6825/3209586931', 'cluster_addr': '172.21.15.153:6827/3209586931', 'heartbeat_back_addr': '172.21.15.153:6831/3209586931', 'heartbeat_front_addr': '172.21.15.153:6829/3209586931', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cc715599-0461-477b-beb2-8777d6f36b22', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 409879231}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 409879231}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 409879231}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 409879231}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 409879231}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 409879231}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 409879231}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 409879231}]}, 'public_addr': '172.21.15.173:6825/409879231', 'cluster_addr': '172.21.15.173:6827/409879231', 'heartbeat_back_addr': '172.21.15.173:6831/409879231', 'heartbeat_front_addr': '172.21.15.173:6829/409879231', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ca5bea1c-f309-4258-b85d-738097037962', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6819', 'nonce': 3465541770}, {'type': 'v1', 'addr': '172.21.15.181:6822', 'nonce': 3465541770}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6826', 'nonce': 3465541770}, {'type': 'v1', 'addr': '172.21.15.181:6827', 'nonce': 3465541770}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6830', 'nonce': 3465541770}, {'type': 'v1', 'addr': '172.21.15.181:6831', 'nonce': 3465541770}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6828', 'nonce': 3465541770}, {'type': 'v1', 'addr': '172.21.15.181:6829', 'nonce': 3465541770}]}, 'public_addr': '172.21.15.181:6822/3465541770', 'cluster_addr': '172.21.15.181:6827/3465541770', 'heartbeat_back_addr': '172.21.15.181:6831/3465541770', 'heartbeat_front_addr': '172.21.15.181:6829/3465541770', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1e80e965-2a46-468c-bcb8-49bee8dd19c1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 1763196448}, {'type': 'v1', 'addr': '172.21.15.190:6803', 'nonce': 1763196448}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 1763196448}, {'type': 'v1', 'addr': '172.21.15.190:6805', 'nonce': 1763196448}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 1763196448}, {'type': 'v1', 'addr': '172.21.15.190:6809', 'nonce': 1763196448}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 1763196448}, {'type': 'v1', 'addr': '172.21.15.190:6807', 'nonce': 1763196448}]}, 'public_addr': '172.21.15.190:6803/1763196448', 'cluster_addr': '172.21.15.190:6805/1763196448', 'heartbeat_back_addr': '172.21.15.190:6809/1763196448', 'heartbeat_front_addr': '172.21.15.190:6807/1763196448', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '87179b74-818f-47df-ad70-1392cfe227e9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6800', 'nonce': 1068520389}, {'type': 'v1', 'addr': '172.21.15.153:6801', 'nonce': 1068520389}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6802', 'nonce': 1068520389}, {'type': 'v1', 'addr': '172.21.15.153:6803', 'nonce': 1068520389}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6806', 'nonce': 1068520389}, {'type': 'v1', 'addr': '172.21.15.153:6807', 'nonce': 1068520389}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6804', 'nonce': 1068520389}, {'type': 'v1', 'addr': '172.21.15.153:6805', 'nonce': 1068520389}]}, 'public_addr': '172.21.15.153:6801/1068520389', 'cluster_addr': '172.21.15.153:6803/1068520389', 'heartbeat_back_addr': '172.21.15.153:6807/1068520389', 'heartbeat_front_addr': '172.21.15.153:6805/1068520389', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '10d3c443-1748-4a7c-8b60-b58cfd565b4d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 210565898}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 210565898}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 210565898}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 210565898}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 210565898}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 210565898}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 210565898}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 210565898}]}, 'public_addr': '172.21.15.173:6801/210565898', 'cluster_addr': '172.21.15.173:6803/210565898', 'heartbeat_back_addr': '172.21.15.173:6807/210565898', 'heartbeat_front_addr': '172.21.15.173:6805/210565898', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '81bb6569-d02b-4638-8b76-f99756d415b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6800', 'nonce': 2007605188}, {'type': 'v1', 'addr': '172.21.15.181:6801', 'nonce': 2007605188}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 2007605188}, {'type': 'v1', 'addr': '172.21.15.181:6803', 'nonce': 2007605188}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 2007605188}, {'type': 'v1', 'addr': '172.21.15.181:6807', 'nonce': 2007605188}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 2007605188}, {'type': 'v1', 'addr': '172.21.15.181:6805', 'nonce': 2007605188}]}, 'public_addr': '172.21.15.181:6801/2007605188', 'cluster_addr': '172.21.15.181:6803/2007605188', 'heartbeat_back_addr': '172.21.15.181:6807/2007605188', 'heartbeat_front_addr': '172.21.15.181:6805/2007605188', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'b8aefe11-845f-48a4-930c-b05cdfbb460a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 2456507663}, {'type': 'v1', 'addr': '172.21.15.190:6812', 'nonce': 2456507663}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 2456507663}, {'type': 'v1', 'addr': '172.21.15.190:6816', 'nonce': 2456507663}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6822', 'nonce': 2456507663}, {'type': 'v1', 'addr': '172.21.15.190:6824', 'nonce': 2456507663}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6818', 'nonce': 2456507663}, {'type': 'v1', 'addr': '172.21.15.190:6820', 'nonce': 2456507663}]}, 'public_addr': '172.21.15.190:6812/2456507663', 'cluster_addr': '172.21.15.190:6816/2456507663', 'heartbeat_back_addr': '172.21.15.190:6824/2456507663', 'heartbeat_front_addr': '172.21.15.190:6820/2456507663', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': '2165437d-26ba-4e50-8646-8cedbc10da53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6808', 'nonce': 819245857}, {'type': 'v1', 'addr': '172.21.15.153:6809', 'nonce': 819245857}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6810', 'nonce': 819245857}, {'type': 'v1', 'addr': '172.21.15.153:6811', 'nonce': 819245857}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6814', 'nonce': 819245857}, {'type': 'v1', 'addr': '172.21.15.153:6815', 'nonce': 819245857}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6812', 'nonce': 819245857}, {'type': 'v1', 'addr': '172.21.15.153:6813', 'nonce': 819245857}]}, 'public_addr': '172.21.15.153:6809/819245857', 'cluster_addr': '172.21.15.153:6811/819245857', 'heartbeat_back_addr': '172.21.15.153:6815/819245857', 'heartbeat_front_addr': '172.21.15.153:6813/819245857', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': '7b9f38b4-f5ff-43f1-a373-2aaed710026d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 3482699195}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 3482699195}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 3482699195}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 3482699195}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 3482699195}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 3482699195}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 3482699195}, {'type': 'v1', 'addr': '172.21.15.173:6814', 'nonce': 3482699195}]}, 'public_addr': '172.21.15.173:6809/3482699195', 'cluster_addr': '172.21.15.173:6811/3482699195', 'heartbeat_back_addr': '172.21.15.173:6819/3482699195', 'heartbeat_front_addr': '172.21.15.173:6814/3482699195', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': '054b27d2-8a4d-4851-8af8-78b71e6a537e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 2780345165}, {'type': 'v1', 'addr': '172.21.15.181:6809', 'nonce': 2780345165}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 2780345165}, {'type': 'v1', 'addr': '172.21.15.181:6811', 'nonce': 2780345165}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 2780345165}, {'type': 'v1', 'addr': '172.21.15.181:6815', 'nonce': 2780345165}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 2780345165}, {'type': 'v1', 'addr': '172.21.15.181:6813', 'nonce': 2780345165}]}, 'public_addr': '172.21.15.181:6809/2780345165', 'cluster_addr': '172.21.15.181:6811/2780345165', 'heartbeat_back_addr': '172.21.15.181:6815/2780345165', 'heartbeat_front_addr': '172.21.15.181:6813/2780345165', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': 'e650f936-3712-47cd-9b17-d732991a27a3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6811', 'nonce': 3871145029}, {'type': 'v1', 'addr': '172.21.15.190:6813', 'nonce': 3871145029}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6815', 'nonce': 3871145029}, {'type': 'v1', 'addr': '172.21.15.190:6817', 'nonce': 3871145029}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6823', 'nonce': 3871145029}, {'type': 'v1', 'addr': '172.21.15.190:6825', 'nonce': 3871145029}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6819', 'nonce': 3871145029}, {'type': 'v1', 'addr': '172.21.15.190:6821', 'nonce': 3871145029}]}, 'public_addr': '172.21.15.190:6813/3871145029', 'cluster_addr': '172.21.15.190:6817/3871145029', 'heartbeat_back_addr': '172.21.15.190:6825/3871145029', 'heartbeat_front_addr': '172.21.15.190:6821/3871145029', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-01T04:15:54.898068+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-01T04:15:54.918555+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-01T04:15:55.123290+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-01T04:15:54.920765+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-01T04:15:54.941559+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-01T04:15:55.003678+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-01T04:15:54.002527+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-01T04:15:55.028107+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-01T04:15:54.920344+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-01T04:15:54.893447+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-01T04:15:55.013907+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-01T04:15:54.898025+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-01T04:15:54.650893+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-01T04:15:55.003356+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-01T04:15:55.008057+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-11-01T04:15:55.065472+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'shecprofile': {'c': '2', 'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'k': '4', 'm': '3', 'name': 'shecprofile', 'plugin': 'shec', 'technique': 'multiple'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-11-01T04:16:12.776 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-01T04:16:12.776 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:12.776 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-01T04:16:12.777 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:12.777 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-01T04:16:12.778 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:12.779 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-01T04:16:12.779 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:12.769+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:12.871 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:12.869+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:12.907 INFO:tasks.rados.rados.0.smithi190.stdout:1: write initial oid 1 2023-11-01T04:16:12.907 INFO:tasks.rados.rados.0.smithi190.stdout:1: seq_num 0 ranges {0=32768} 2023-11-01T04:16:12.908 INFO:tasks.rados.rados.0.smithi190.stdout:1: writing smithi19039307-1 from 0 to 32768 tid 1 2023-11-01T04:16:12.908 INFO:tasks.rados.rados.0.smithi190.stdout:2: write initial oid 2 2023-11-01T04:16:12.909 INFO:tasks.rados.rados.0.smithi190.stdout:2: seq_num 1 ranges {0=32768} 2023-11-01T04:16:12.909 INFO:tasks.rados.rados.0.smithi190.stdout:2: writing smithi19039307-2 from 0 to 32768 tid 1 2023-11-01T04:16:12.909 INFO:tasks.rados.rados.0.smithi190.stdout:3: write initial oid 3 2023-11-01T04:16:12.909 INFO:tasks.rados.rados.0.smithi190.stdout:3: seq_num 2 ranges {0=49152} 2023-11-01T04:16:12.910 INFO:tasks.rados.rados.0.smithi190.stdout:3: writing smithi19039307-3 from 0 to 49152 tid 1 2023-11-01T04:16:12.910 INFO:tasks.rados.rados.0.smithi190.stdout:4: write initial oid 4 2023-11-01T04:16:12.910 INFO:tasks.rados.rados.0.smithi190.stdout:4: seq_num 3 ranges {0=65536} 2023-11-01T04:16:12.911 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:12.911 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:12.912 INFO:tasks.rados.rados.0.smithi190.stdout:4: writing smithi19039307-4 from 0 to 65536 tid 1 2023-11-01T04:16:12.912 INFO:tasks.rados.rados.0.smithi190.stdout:5: write initial oid 5 2023-11-01T04:16:12.913 INFO:tasks.rados.rados.0.smithi190.stdout:5: seq_num 4 ranges {0=81920} 2023-11-01T04:16:12.913 INFO:tasks.rados.rados.0.smithi190.stdout:5: writing smithi19039307-5 from 0 to 81920 tid 1 2023-11-01T04:16:12.913 INFO:tasks.rados.rados.0.smithi190.stdout:6: write initial oid 6 2023-11-01T04:16:12.913 INFO:tasks.rados.rados.0.smithi190.stdout:6: seq_num 5 ranges {0=98304} 2023-11-01T04:16:12.913 INFO:tasks.rados.rados.0.smithi190.stdout:6: writing smithi19039307-6 from 0 to 98304 tid 1 2023-11-01T04:16:12.913 INFO:tasks.rados.rados.0.smithi190.stdout:7: write initial oid 7 2023-11-01T04:16:12.914 INFO:tasks.rados.rados.0.smithi190.stdout:7: seq_num 6 ranges {0=114688} 2023-11-01T04:16:12.914 INFO:tasks.rados.rados.0.smithi190.stdout:7: writing smithi19039307-7 from 0 to 114688 tid 1 2023-11-01T04:16:12.914 INFO:tasks.rados.rados.0.smithi190.stdout:8: write initial oid 8 2023-11-01T04:16:12.914 INFO:tasks.rados.rados.0.smithi190.stdout:8: seq_num 7 ranges {0=131072} 2023-11-01T04:16:12.915 INFO:tasks.rados.rados.0.smithi190.stdout:8: writing smithi19039307-8 from 0 to 131072 tid 1 2023-11-01T04:16:12.915 INFO:tasks.rados.rados.0.smithi190.stdout:9: write initial oid 9 2023-11-01T04:16:12.915 INFO:tasks.rados.rados.0.smithi190.stdout:9: seq_num 8 ranges {0=147456} 2023-11-01T04:16:12.916 INFO:tasks.rados.rados.0.smithi190.stdout:9: writing smithi19039307-9 from 0 to 147456 tid 1 2023-11-01T04:16:12.916 INFO:tasks.rados.rados.0.smithi190.stdout:10: write initial oid 10 2023-11-01T04:16:12.916 INFO:tasks.rados.rados.0.smithi190.stdout:10: seq_num 9 ranges {0=163840} 2023-11-01T04:16:12.916 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-01T04:16:12.917 INFO:tasks.rados.rados.0.smithi190.stdout:10: writing smithi19039307-10 from 0 to 163840 tid 1 2023-11-01T04:16:12.917 INFO:tasks.rados.rados.0.smithi190.stdout:11: write initial oid 11 2023-11-01T04:16:12.918 INFO:tasks.rados.rados.0.smithi190.stdout:11: seq_num 10 ranges {0=180224} 2023-11-01T04:16:12.918 INFO:tasks.rados.rados.0.smithi190.stdout:11: writing smithi19039307-11 from 0 to 180224 tid 1 2023-11-01T04:16:12.919 INFO:tasks.rados.rados.0.smithi190.stdout:12: write initial oid 12 2023-11-01T04:16:12.919 INFO:tasks.rados.rados.0.smithi190.stdout:12: seq_num 11 ranges {0=196608} 2023-11-01T04:16:12.920 INFO:tasks.rados.rados.0.smithi190.stdout:12: writing smithi19039307-12 from 0 to 196608 tid 1 2023-11-01T04:16:12.921 INFO:tasks.rados.rados.0.smithi190.stdout:13: write initial oid 13 2023-11-01T04:16:12.921 INFO:tasks.rados.rados.0.smithi190.stdout:13: seq_num 12 ranges {0=212992} 2023-11-01T04:16:12.923 INFO:tasks.rados.rados.0.smithi190.stdout:13: writing smithi19039307-13 from 0 to 212992 tid 1 2023-11-01T04:16:12.923 INFO:tasks.rados.rados.0.smithi190.stdout:14: write initial oid 14 2023-11-01T04:16:12.923 INFO:tasks.rados.rados.0.smithi190.stdout:14: seq_num 13 ranges {0=229376} 2023-11-01T04:16:12.925 INFO:tasks.rados.rados.0.smithi190.stdout:14: writing smithi19039307-14 from 0 to 229376 tid 1 2023-11-01T04:16:12.925 INFO:tasks.rados.rados.0.smithi190.stdout:15: write initial oid 15 2023-11-01T04:16:12.925 INFO:tasks.rados.rados.0.smithi190.stdout:15: seq_num 14 ranges {0=245760} 2023-11-01T04:16:12.927 INFO:tasks.rados.rados.0.smithi190.stdout:15: writing smithi19039307-15 from 0 to 245760 tid 1 2023-11-01T04:16:12.927 INFO:tasks.rados.rados.0.smithi190.stdout:16: write initial oid 16 2023-11-01T04:16:12.927 INFO:tasks.rados.rados.0.smithi190.stdout:16: seq_num 15 ranges {0=262144} 2023-11-01T04:16:12.928 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-11-01T04:16:12.929 INFO:tasks.rados.rados.0.smithi190.stdout:16: writing smithi19039307-16 from 0 to 262144 tid 1 2023-11-01T04:16:12.930 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:12.975 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:12.976+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:13.017 INFO:tasks.rados.rados.0.smithi190.stdout:1: finishing write tid 1 to smithi19039307-1 2023-11-01T04:16:13.052 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:13.053 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:13.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-11-01T04:16:13.073 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:13.069+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:13.121 INFO:tasks.rados.rados.0.smithi190.stdout:8: finishing write tid 1 to smithi19039307-8 2023-11-01T04:16:13.132 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:16:13.133 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-11-01T04:16:13.150 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-11-01T04:16:12.904789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":134,"num_read_kb":116,"num_write":200,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":73,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":439764,"kb_used_data":6616,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509509676,"statfs":{"total":1546188226560,"available":1545737908224,"internally_reserved":0,"allocated":6774784,"data_stored":3529892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25505,"internal_metadata":442473567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":27,"num_write":58,"num_write_kb":1306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.034113"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-01T04:16:11.274054+0000","last_change":"2023-11-01T04:16:11.273007+0000","last_active":"2023-11-01T04:16:11.274054+0000","last_peered":"2023-11-01T04:16:11.274054+0000","last_clean":"2023-11-01T04:16:11.274054+0000","last_became_active":"2023-11-01T04:16:11.171434+0000","last_became_peered":"2023-11-01T04:16:11.171434+0000","last_unstale":"2023-11-01T04:16:11.274054+0000","last_undegraded":"2023-11-01T04:16:11.274054+0000","last_fullsized":"2023-11-01T04:16:11.274054+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:29.251391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,15,13,14,11,2],"acting":[0,3,15,13,14,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-01T04:16:12.002741+0000","last_change":"2023-11-01T04:16:11.270084+0000","last_active":"2023-11-01T04:16:12.002741+0000","last_peered":"2023-11-01T04:16:12.002741+0000","last_clean":"2023-11-01T04:16:12.002741+0000","last_became_active":"2023-11-01T04:16:11.168562+0000","last_became_peered":"2023-11-01T04:16:11.168562+0000","last_unstale":"2023-11-01T04:16:12.002741+0000","last_undegraded":"2023-11-01T04:16:12.002741+0000","last_fullsized":"2023-11-01T04:16:12.002741+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:23.071460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1,4,2,3,5],"acting":[11,9,1,4,2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-01T04:16:12.172901+0000","last_change":"2023-11-01T04:16:11.269767+0000","last_active":"2023-11-01T04:16:12.172901+0000","last_peered":"2023-11-01T04:16:12.172901+0000","last_clean":"2023-11-01T04:16:12.172901+0000","last_became_active":"2023-11-01T04:16:11.267819+0000","last_became_peered":"2023-11-01T04:16:11.267819+0000","last_unstale":"2023-11-01T04:16:12.172901+0000","last_undegraded":"2023-11-01T04:16:12.172901+0000","last_fullsized":"2023-11-01T04:16:12.172901+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:11.637086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,11,0,8,2,6],"acting":[12,4,11,0,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-01T04:16:11.372123+0000","last_change":"2023-11-01T04:16:11.271347+0000","last_active":"2023-11-01T04:16:11.372123+0000","last_peered":"2023-11-01T04:16:11.372123+0000","last_clean":"2023-11-01T04:16:11.372123+0000","last_became_active":"2023-11-01T04:16:11.169936+0000","last_became_peered":"2023-11-01T04:16:11.169936+0000","last_unstale":"2023-11-01T04:16:11.372123+0000","last_undegraded":"2023-11-01T04:16:11.372123+0000","last_fullsized":"2023-11-01T04:16:11.372123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:16.135815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,12,3,7,2,6],"acting":[0,10,12,3,7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-01T04:16:11.430991+0000","last_change":"2023-11-01T04:16:11.430151+0000","last_active":"2023-11-01T04:16:11.430991+0000","last_peered":"2023-11-01T04:16:11.430991+0000","last_clean":"2023-11-01T04:16:11.430991+0000","last_became_active":"2023-11-01T04:16:11.228189+0000","last_became_peered":"2023-11-01T04:16:11.228189+0000","last_unstale":"2023-11-01T04:16:11.430991+0000","last_undegraded":"2023-11-01T04:16:11.430991+0000","last_fullsized":"2023-11-01T04:16:11.430991+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:17.465687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1,9,11,4,14],"acting":[10,6,1,9,11,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-01T04:16:12.172907+0000","last_change":"2023-11-01T04:16:11.104438+0000","last_active":"2023-11-01T04:16:12.172907+0000","last_peered":"2023-11-01T04:16:12.172907+0000","last_clean":"2023-11-01T04:16:12.172907+0000","last_became_active":"2023-11-01T04:16:11.103812+0000","last_became_peered":"2023-11-01T04:16:11.103812+0000","last_unstale":"2023-11-01T04:16:12.172907+0000","last_undegraded":"2023-11-01T04:16:12.172907+0000","last_fullsized":"2023-11-01T04:16:12.172907+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:37.382497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,1,9,3,15,8],"acting":[12,4,1,9,3,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-01T04:16:12.102188+0000","last_change":"2023-11-01T04:16:11.197469+0000","last_active":"2023-11-01T04:16:12.102188+0000","last_peered":"2023-11-01T04:16:12.102188+0000","last_clean":"2023-11-01T04:16:12.102188+0000","last_became_active":"2023-11-01T04:16:10.995421+0000","last_became_peered":"2023-11-01T04:16:10.995421+0000","last_unstale":"2023-11-01T04:16:12.102188+0000","last_undegraded":"2023-11-01T04:16:12.102188+0000","last_fullsized":"2023-11-01T04:16:12.102188+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:23.925851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,14,11,4,10,3],"acting":[7,1,14,11,4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-01T04:16:12.000359+0000","last_change":"2023-11-01T04:16:11.115655+0000","last_active":"2023-11-01T04:16:12.000359+0000","last_peered":"2023-11-01T04:16:12.000359+0000","last_clean":"2023-11-01T04:16:12.000359+0000","last_became_active":"2023-11-01T04:16:11.114141+0000","last_became_peered":"2023-11-01T04:16:11.114141+0000","last_unstale":"2023-11-01T04:16:12.000359+0000","last_undegraded":"2023-11-01T04:16:12.000359+0000","last_fullsized":"2023-11-01T04:16:12.000359+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:23.071460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,11,3,14,0,7],"acting":[5,10,11,3,14,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"33'155","reported_seq":213,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-01T04:16:12.620241+0000","last_change":"2023-11-01T04:15:59.016143+0000","last_active":"2023-11-01T04:16:12.620241+0000","last_peered":"2023-11-01T04:16:12.620241+0000","last_clean":"2023-11-01T04:16:12.620241+0000","last_became_active":"2023-11-01T04:15:59.014137+0000","last_became_peered":"2023-11-01T04:15:59.014137+0000","last_unstale":"2023-11-01T04:16:12.620241+0000","last_undegraded":"2023-11-01T04:16:12.620241+0000","last_fullsized":"2023-11-01T04:16:12.620241+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_clean_scrub_stamp":"2023-11-01T04:15:57.686019+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:15.188172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":134,"num_read_kb":116,"num_write":200,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-01T04:16:10.898411+0000","last_change":"2023-11-01T04:16:02.077904+0000","last_active":"2023-11-01T04:16:10.898411+0000","last_peered":"2023-11-01T04:16:10.898411+0000","last_clean":"2023-11-01T04:16:10.898411+0000","last_became_active":"2023-11-01T04:16:02.077101+0000","last_became_peered":"2023-11-01T04:16:02.077101+0000","last_unstale":"2023-11-01T04:16:10.898411+0000","last_undegraded":"2023-11-01T04:16:10.898411+0000","last_fullsized":"2023-11-01T04:16:10.898411+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:10.452369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-01T04:16:12.104889+0000","last_change":"2023-11-01T04:16:11.200968+0000","last_active":"2023-11-01T04:16:12.104889+0000","last_peered":"2023-11-01T04:16:12.104889+0000","last_clean":"2023-11-01T04:16:12.104889+0000","last_became_active":"2023-11-01T04:16:11.100400+0000","last_became_peered":"2023-11-01T04:16:11.100400+0000","last_unstale":"2023-11-01T04:16:12.104889+0000","last_undegraded":"2023-11-01T04:16:12.104889+0000","last_fullsized":"2023-11-01T04:16:12.104889+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:32.836167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,3,0,4,8,1,7],"acting":[13,3,0,4,8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-01T04:16:10.897320+0000","last_change":"2023-11-01T04:16:01.930714+0000","last_active":"2023-11-01T04:16:10.897320+0000","last_peered":"2023-11-01T04:16:10.897320+0000","last_clean":"2023-11-01T04:16:10.897320+0000","last_became_active":"2023-11-01T04:16:01.930333+0000","last_became_peered":"2023-11-01T04:16:01.930333+0000","last_unstale":"2023-11-01T04:16:10.897320+0000","last_undegraded":"2023-11-01T04:16:10.897320+0000","last_fullsized":"2023-11-01T04:16:10.897320+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:06.734556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-01T04:16:11.198952+0000","last_change":"2023-11-01T04:16:11.198408+0000","last_active":"2023-11-01T04:16:11.198952+0000","last_peered":"2023-11-01T04:16:11.198952+0000","last_clean":"2023-11-01T04:16:11.198952+0000","last_became_active":"2023-11-01T04:16:11.197508+0000","last_became_peered":"2023-11-01T04:16:11.197508+0000","last_unstale":"2023-11-01T04:16:11.198952+0000","last_undegraded":"2023-11-01T04:16:11.198952+0000","last_fullsized":"2023-11-01T04:16:11.198952+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:30.430777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9,5,7,3,13],"acting":[6,4,9,5,7,3,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-01T04:16:10.891082+0000","last_change":"2023-11-01T04:16:01.757411+0000","last_active":"2023-11-01T04:16:10.891082+0000","last_peered":"2023-11-01T04:16:10.891082+0000","last_clean":"2023-11-01T04:16:10.891082+0000","last_became_active":"2023-11-01T04:16:01.756620+0000","last_became_peered":"2023-11-01T04:16:01.756620+0000","last_unstale":"2023-11-01T04:16:10.891082+0000","last_undegraded":"2023-11-01T04:16:10.891082+0000","last_fullsized":"2023-11-01T04:16:10.891082+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:02.829288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-01T04:16:12.001481+0000","last_change":"2023-11-01T04:16:11.015551+0000","last_active":"2023-11-01T04:16:12.001481+0000","last_peered":"2023-11-01T04:16:12.001481+0000","last_clean":"2023-11-01T04:16:12.001481+0000","last_became_active":"2023-11-01T04:16:11.014296+0000","last_became_peered":"2023-11-01T04:16:11.014296+0000","last_unstale":"2023-11-01T04:16:12.001481+0000","last_undegraded":"2023-11-01T04:16:12.001481+0000","last_fullsized":"2023-11-01T04:16:12.001481+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:28.660261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4,1,5,15,6],"acting":[3,7,4,1,5,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-01T04:16:10.897304+0000","last_change":"2023-11-01T04:16:02.153588+0000","last_active":"2023-11-01T04:16:10.897304+0000","last_peered":"2023-11-01T04:16:10.897304+0000","last_clean":"2023-11-01T04:16:10.897304+0000","last_became_active":"2023-11-01T04:16:02.053124+0000","last_became_peered":"2023-11-01T04:16:02.053124+0000","last_unstale":"2023-11-01T04:16:10.897304+0000","last_undegraded":"2023-11-01T04:16:10.897304+0000","last_fullsized":"2023-11-01T04:16:10.897304+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:11.022108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-01T04:16:12.104946+0000","last_change":"2023-11-01T04:16:11.072210+0000","last_active":"2023-11-01T04:16:12.104946+0000","last_peered":"2023-11-01T04:16:12.104946+0000","last_clean":"2023-11-01T04:16:12.104946+0000","last_became_active":"2023-11-01T04:16:11.070168+0000","last_became_peered":"2023-11-01T04:16:11.070168+0000","last_unstale":"2023-11-01T04:16:12.104946+0000","last_undegraded":"2023-11-01T04:16:12.104946+0000","last_fullsized":"2023-11-01T04:16:12.104946+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:24.861708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8,2,0,11,5,12],"acting":[13,8,2,0,11,5,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-01T04:16:12.101171+0000","last_change":"2023-11-01T04:16:11.269432+0000","last_active":"2023-11-01T04:16:12.101171+0000","last_peered":"2023-11-01T04:16:12.101171+0000","last_clean":"2023-11-01T04:16:12.101171+0000","last_became_active":"2023-11-01T04:16:11.168020+0000","last_became_peered":"2023-11-01T04:16:11.168020+0000","last_unstale":"2023-11-01T04:16:12.101171+0000","last_undegraded":"2023-11-01T04:16:12.101171+0000","last_fullsized":"2023-11-01T04:16:12.101171+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:25.697185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,14,13,10,2,11],"acting":[8,6,14,13,10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-01T04:16:09.903293+0000","last_change":"2023-11-01T04:16:01.756078+0000","last_active":"2023-11-01T04:16:09.903293+0000","last_peered":"2023-11-01T04:16:09.903293+0000","last_clean":"2023-11-01T04:16:09.903293+0000","last_became_active":"2023-11-01T04:16:01.755533+0000","last_became_peered":"2023-11-01T04:16:01.755533+0000","last_unstale":"2023-11-01T04:16:09.903293+0000","last_undegraded":"2023-11-01T04:16:09.903293+0000","last_fullsized":"2023-11-01T04:16:09.903293+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:28.274958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-01T04:16:11.208639+0000","last_change":"2023-11-01T04:16:11.208052+0000","last_active":"2023-11-01T04:16:11.208639+0000","last_peered":"2023-11-01T04:16:11.208639+0000","last_clean":"2023-11-01T04:16:11.208639+0000","last_became_active":"2023-11-01T04:16:11.207236+0000","last_became_peered":"2023-11-01T04:16:11.207236+0000","last_unstale":"2023-11-01T04:16:11.208639+0000","last_undegraded":"2023-11-01T04:16:11.208639+0000","last_fullsized":"2023-11-01T04:16:11.208639+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:15.863391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,15,14,1,6,3,13],"acting":[10,15,14,1,6,3,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-01T04:16:10.894550+0000","last_change":"2023-11-01T04:16:02.168355+0000","last_active":"2023-11-01T04:16:10.894550+0000","last_peered":"2023-11-01T04:16:10.894550+0000","last_clean":"2023-11-01T04:16:10.894550+0000","last_became_active":"2023-11-01T04:16:02.167473+0000","last_became_peered":"2023-11-01T04:16:02.167473+0000","last_unstale":"2023-11-01T04:16:10.894550+0000","last_undegraded":"2023-11-01T04:16:10.894550+0000","last_fullsized":"2023-11-01T04:16:10.894550+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:10.452369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-01T04:16:11.999264+0000","last_change":"2023-11-01T04:16:11.305991+0000","last_active":"2023-11-01T04:16:11.999264+0000","last_peered":"2023-11-01T04:16:11.999264+0000","last_clean":"2023-11-01T04:16:11.999264+0000","last_became_active":"2023-11-01T04:16:11.205734+0000","last_became_peered":"2023-11-01T04:16:11.205734+0000","last_unstale":"2023-11-01T04:16:11.999264+0000","last_undegraded":"2023-11-01T04:16:11.999264+0000","last_fullsized":"2023-11-01T04:16:11.999264+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:27.392401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,9,7,2,5,0],"acting":[14,11,9,7,2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-01T04:16:10.895085+0000","last_change":"2023-11-01T04:16:01.875964+0000","last_active":"2023-11-01T04:16:10.895085+0000","last_peered":"2023-11-01T04:16:10.895085+0000","last_clean":"2023-11-01T04:16:10.895085+0000","last_became_active":"2023-11-01T04:16:01.875255+0000","last_became_peered":"2023-11-01T04:16:01.875255+0000","last_unstale":"2023-11-01T04:16:10.895085+0000","last_undegraded":"2023-11-01T04:16:10.895085+0000","last_fullsized":"2023-11-01T04:16:10.895085+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:05.611210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11],"acting":[14,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-01T04:16:11.999193+0000","last_change":"2023-11-01T04:16:11.105212+0000","last_active":"2023-11-01T04:16:11.999193+0000","last_peered":"2023-11-01T04:16:11.999193+0000","last_clean":"2023-11-01T04:16:11.999193+0000","last_became_active":"2023-11-01T04:16:11.103960+0000","last_became_peered":"2023-11-01T04:16:11.103960+0000","last_unstale":"2023-11-01T04:16:11.999193+0000","last_undegraded":"2023-11-01T04:16:11.999193+0000","last_fullsized":"2023-11-01T04:16:11.999193+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:36.644129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,1,8,12,0,9],"acting":[14,13,1,8,12,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-01T04:16:10.897343+0000","last_change":"2023-11-01T04:16:01.855207+0000","last_active":"2023-11-01T04:16:10.897343+0000","last_peered":"2023-11-01T04:16:10.897343+0000","last_clean":"2023-11-01T04:16:10.897343+0000","last_became_active":"2023-11-01T04:16:01.854811+0000","last_became_peered":"2023-11-01T04:16:01.854811+0000","last_unstale":"2023-11-01T04:16:10.897343+0000","last_undegraded":"2023-11-01T04:16:10.897343+0000","last_fullsized":"2023-11-01T04:16:10.897343+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:08.297413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":134,"num_read_kb":116,"num_write":200,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28228,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343612,"statfs":{"total":96636764160,"available":96607858688,"internally_reserved":0,"allocated":847872,"data_stored":637276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,6,7,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":109.473}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":109.26600000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":109.429}]}]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":159647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,5,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.355}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.494}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.536}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.464}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3260000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]}]},{"osd":13,"up_from":22,"seq":94489280518,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":831488,"data_stored":627135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.121}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4199999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5700000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.532}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.601}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7130000000000001}]}]},{"osd":6,"up_from":21,"seq":90194313223,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":360448,"data_stored":159647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[3,4,5,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.006}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]}]},{"osd":0,"up_from":22,"seq":94489280518,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":159647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]}]},{"osd":1,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":360448,"data_stored":159647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.452}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.605}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.649}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8320000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1840000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8799999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9530000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0070000000000001}]}]},{"osd":2,"up_from":22,"seq":94489280519,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":159647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]},{"osd":3,"up_from":22,"seq":94489280518,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":169788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.302}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.345}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6859999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.431}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7410000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5620000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4770000000000001}]}]},{"osd":4,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":376832,"data_stored":169788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,5,6,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]}]},{"osd":5,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":159647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.377}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.131}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.274}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.351}]}]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":159647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,6,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.256}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1839999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2969999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.669}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.702}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.585}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5129999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6279999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280519,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":360448,"data_stored":159647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4139999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2689999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.375}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4610000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.403}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.502}]}]},{"osd":9,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":360448,"data_stored":159647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,3,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.266}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9399999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.363}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4550000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.093}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.498}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]}]},{"osd":10,"up_from":22,"seq":94489280518,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":159647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1140000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.034}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1919999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1739999999999999}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]}]},{"osd":11,"up_from":22,"seq":94489280517,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":159647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.766}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4610000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8919999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.208}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5669999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.613}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.028}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8420000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.077}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9399999999999999}]}]},{"osd":12,"up_from":22,"seq":94489280519,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":169788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]}]}],"pool_statfs":[{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-01T04:16:13.153 INFO:tasks.thrashosds.thrasher:Setting 2.4 to [9, 8] 2023-11-01T04:16:13.154 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.4', '9', '8'] 2023-11-01T04:16:13.154 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.4 9 8 2023-11-01T04:16:13.174 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:13.174+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:13.218 INFO:tasks.rados.rados.0.smithi190.stdout:13: finishing write tid 1 to smithi19039307-13 2023-11-01T04:16:13.219 INFO:tasks.rados.rados.0.smithi190.stdout:13: finishing write tid 2 to smithi19039307-13 2023-11-01T04:16:13.220 INFO:tasks.rados.rados.0.smithi190.stdout:10: finishing write tid 1 to smithi19039307-10 2023-11-01T04:16:13.221 INFO:tasks.rados.rados.0.smithi190.stdout:8: finishing write tid 2 to smithi19039307-8 2023-11-01T04:16:13.224 INFO:tasks.rados.rados.0.smithi190.stdout:9: finishing write tid 1 to smithi19039307-9 2023-11-01T04:16:13.224 INFO:tasks.rados.rados.0.smithi190.stdout:3: finishing write tid 1 to smithi19039307-3 2023-11-01T04:16:13.225 INFO:tasks.rados.rados.0.smithi190.stdout:3: finishing write tid 2 to smithi19039307-3 2023-11-01T04:16:13.225 INFO:tasks.rados.rados.0.smithi190.stdout:7: finishing write tid 1 to smithi19039307-7 2023-11-01T04:16:13.225 INFO:tasks.rados.rados.0.smithi190.stdout:5: finishing write tid 1 to smithi19039307-5 2023-11-01T04:16:13.226 INFO:tasks.rados.rados.0.smithi190.stdout:5: finishing write tid 2 to smithi19039307-5 2023-11-01T04:16:13.227 INFO:tasks.rados.rados.0.smithi190.stdout:7: finishing write tid 2 to smithi19039307-7 2023-11-01T04:16:13.236 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:13.237 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:13.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-01T04:16:13.295 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:16:13.296+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:16:13.316 INFO:tasks.rados.rados.0.smithi190.stdout:6: finishing write tid 1 to smithi19039307-6 2023-11-01T04:16:13.317 INFO:tasks.rados.rados.0.smithi190.stdout:10: finishing write tid 2 to smithi19039307-10 2023-11-01T04:16:13.317 INFO:tasks.rados.rados.0.smithi190.stdout:6: finishing write tid 2 to smithi19039307-6 2023-11-01T04:16:13.330 INFO:tasks.rados.rados.0.smithi190.stdout:1: finishing write tid 2 to smithi19039307-1 2023-11-01T04:16:13.332 INFO:tasks.rados.rados.0.smithi190.stdout:4: finishing write tid 1 to smithi19039307-4 2023-11-01T04:16:13.332 INFO:tasks.rados.rados.0.smithi190.stdout:4: finishing write tid 2 to smithi19039307-4 2023-11-01T04:16:13.376 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:13.373+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:13.413 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:13.414 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:13.423 INFO:tasks.rados.rados.0.smithi190.stdout:9: finishing write tid 2 to smithi19039307-9 2023-11-01T04:16:13.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-01T04:16:13.514 INFO:tasks.rados.rados.0.smithi190.stdout:2: finishing write tid 1 to smithi19039307-2 2023-11-01T04:16:13.515 INFO:tasks.rados.rados.0.smithi190.stdout:2: finishing write tid 2 to smithi19039307-2 2023-11-01T04:16:13.515 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:16:13.512+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:16:13.516 INFO:tasks.rados.rados.0.smithi190.stdout:16: finishing write tid 1 to smithi19039307-16 2023-11-01T04:16:13.516 INFO:tasks.rados.rados.0.smithi190.stdout:16: finishing write tid 2 to smithi19039307-16 2023-11-01T04:16:13.524 INFO:tasks.rados.rados.0.smithi190.stdout:8: finishing write tid 3 to smithi19039307-8 2023-11-01T04:16:13.525 INFO:tasks.rados.rados.0.smithi190.stdout:8: oid 8 updating version 0 to 1 2023-11-01T04:16:13.525 INFO:tasks.rados.rados.0.smithi190.stdout:8: oid 8 updating version 1 to 2 2023-11-01T04:16:13.525 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2023-11-01T04:16:13.525 INFO:tasks.rados.rados.0.smithi190.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2023-11-01T04:16:13.525 INFO:tasks.rados.rados.0.smithi190.stdout:8: done (15 left) 2023-11-01T04:16:13.526 INFO:tasks.rados.rados.0.smithi190.stdout:17: write initial oid 17 2023-11-01T04:16:13.526 INFO:tasks.rados.rados.0.smithi190.stdout:17: seq_num 16 ranges {0=278528} 2023-11-01T04:16:13.527 INFO:tasks.rados.rados.0.smithi190.stdout:17: writing smithi19039307-17 from 0 to 278528 tid 1 2023-11-01T04:16:13.527 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:13.533 INFO:tasks.rados.rados.0.smithi190.stdout:7: finishing write tid 3 to smithi19039307-7 2023-11-01T04:16:13.534 INFO:tasks.rados.rados.0.smithi190.stdout:7: oid 7 updating version 0 to 1 2023-11-01T04:16:13.534 INFO:tasks.rados.rados.0.smithi190.stdout:7: oid 7 updating version 1 to 2 2023-11-01T04:16:13.534 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 7 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2023-11-01T04:16:13.534 INFO:tasks.rados.rados.0.smithi190.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2023-11-01T04:16:13.535 INFO:tasks.rados.rados.0.smithi190.stdout:7: done (15 left) 2023-11-01T04:16:13.535 INFO:tasks.rados.rados.0.smithi190.stdout:18: write initial oid 18 2023-11-01T04:16:13.535 INFO:tasks.rados.rados.0.smithi190.stdout:18: seq_num 17 ranges {0=294912} 2023-11-01T04:16:13.536 INFO:tasks.rados.rados.0.smithi190.stdout:18: writing smithi19039307-18 from 0 to 294912 tid 1 2023-11-01T04:16:13.536 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:13.536 INFO:tasks.rados.rados.0.smithi190.stdout:6: finishing write tid 3 to smithi19039307-6 2023-11-01T04:16:13.537 INFO:tasks.rados.rados.0.smithi190.stdout:6: oid 6 updating version 0 to 1 2023-11-01T04:16:13.537 INFO:tasks.rados.rados.0.smithi190.stdout:6: oid 6 updating version 1 to 2 2023-11-01T04:16:13.537 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2023-11-01T04:16:13.537 INFO:tasks.rados.rados.0.smithi190.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2023-11-01T04:16:13.538 INFO:tasks.rados.rados.0.smithi190.stdout:3: finishing write tid 3 to smithi19039307-3 2023-11-01T04:16:13.538 INFO:tasks.rados.rados.0.smithi190.stdout:3: oid 3 updating version 0 to 1 2023-11-01T04:16:13.538 INFO:tasks.rados.rados.0.smithi190.stdout:3: oid 3 updating version 1 to 2 2023-11-01T04:16:13.538 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2023-11-01T04:16:13.539 INFO:tasks.rados.rados.0.smithi190.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2023-11-01T04:16:13.539 INFO:tasks.rados.rados.0.smithi190.stdout:15: finishing write tid 1 to smithi19039307-15 2023-11-01T04:16:13.539 INFO:tasks.rados.rados.0.smithi190.stdout:3: done (15 left) 2023-11-01T04:16:13.539 INFO:tasks.rados.rados.0.smithi190.stdout:6: done (14 left) 2023-11-01T04:16:13.539 INFO:tasks.rados.rados.0.smithi190.stdout:19: write initial oid 19 2023-11-01T04:16:13.540 INFO:tasks.rados.rados.0.smithi190.stdout:19: seq_num 18 ranges {0=311296} 2023-11-01T04:16:13.541 INFO:tasks.rados.rados.0.smithi190.stdout:19: writing smithi19039307-19 from 0 to 311296 tid 1 2023-11-01T04:16:13.541 INFO:tasks.rados.rados.0.smithi190.stdout:20: write initial oid 20 2023-11-01T04:16:13.541 INFO:tasks.rados.rados.0.smithi190.stdout:20: seq_num 19 ranges {0=327680} 2023-11-01T04:16:13.541 INFO:tasks.rados.rados.0.smithi190.stdout:20: writing smithi19039307-20 from 0 to 327680 tid 1 2023-11-01T04:16:13.542 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:13.542 INFO:tasks.rados.rados.0.smithi190.stdout:12: finishing write tid 1 to smithi19039307-12 2023-11-01T04:16:13.542 INFO:tasks.rados.rados.0.smithi190.stdout:1: finishing write tid 3 to smithi19039307-1 2023-11-01T04:16:13.542 INFO:tasks.rados.rados.0.smithi190.stdout:1: oid 1 updating version 0 to 1 2023-11-01T04:16:13.543 INFO:tasks.rados.rados.0.smithi190.stdout:1: oid 1 updating version 1 to 2 2023-11-01T04:16:13.543 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2023-11-01T04:16:13.543 INFO:tasks.rados.rados.0.smithi190.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2023-11-01T04:16:13.543 INFO:tasks.rados.rados.0.smithi190.stdout:4: finishing write tid 3 to smithi19039307-4 2023-11-01T04:16:13.543 INFO:tasks.rados.rados.0.smithi190.stdout:4: oid 4 updating version 0 to 4 2023-11-01T04:16:13.544 INFO:tasks.rados.rados.0.smithi190.stdout:4: oid 4 version 4 is already newer than 3 2023-11-01T04:16:13.544 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 4 v 4 (ObjNum 3 snap 0 seq_num 3) dirty exists 2023-11-01T04:16:13.544 INFO:tasks.rados.rados.0.smithi190.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2023-11-01T04:16:13.544 INFO:tasks.rados.rados.0.smithi190.stdout:1: done (15 left) 2023-11-01T04:16:13.545 INFO:tasks.rados.rados.0.smithi190.stdout:4: done (14 left) 2023-11-01T04:16:13.545 INFO:tasks.rados.rados.0.smithi190.stdout:21: write initial oid 21 2023-11-01T04:16:13.545 INFO:tasks.rados.rados.0.smithi190.stdout:21: seq_num 20 ranges {0=344064} 2023-11-01T04:16:13.546 INFO:tasks.rados.rados.0.smithi190.stdout:21: writing smithi19039307-21 from 0 to 344064 tid 1 2023-11-01T04:16:13.546 INFO:tasks.rados.rados.0.smithi190.stdout:22: write initial oid 22 2023-11-01T04:16:13.546 INFO:tasks.rados.rados.0.smithi190.stdout:22: seq_num 21 ranges {0=360448} 2023-11-01T04:16:13.547 INFO:tasks.rados.rados.0.smithi190.stdout:22: writing smithi19039307-22 from 0 to 360448 tid 1 2023-11-01T04:16:13.547 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:13.578 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:13.576+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:13.592 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:13.592 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:13.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-01T04:16:13.635 INFO:tasks.rados.rados.0.smithi190.stdout:15: finishing write tid 2 to smithi19039307-15 2023-11-01T04:16:13.679 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:13.676+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:13.723 INFO:tasks.rados.rados.0.smithi190.stdout:13: finishing write tid 3 to smithi19039307-13 2023-11-01T04:16:13.723 INFO:tasks.rados.rados.0.smithi190.stdout:13: oid 13 updating version 0 to 1 2023-11-01T04:16:13.723 INFO:tasks.rados.rados.0.smithi190.stdout:13: oid 13 updating version 1 to 2 2023-11-01T04:16:13.724 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 13 v 2 (ObjNum 12 snap 0 seq_num 12) dirty exists 2023-11-01T04:16:13.724 INFO:tasks.rados.rados.0.smithi190.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2023-11-01T04:16:13.724 INFO:tasks.rados.rados.0.smithi190.stdout:13: done (15 left) 2023-11-01T04:16:13.724 INFO:tasks.rados.rados.0.smithi190.stdout:23: write initial oid 23 2023-11-01T04:16:13.725 INFO:tasks.rados.rados.0.smithi190.stdout:23: seq_num 22 ranges {0=376832} 2023-11-01T04:16:13.726 INFO:tasks.rados.rados.0.smithi190.stdout:23: writing smithi19039307-23 from 0 to 376832 tid 1 2023-11-01T04:16:13.726 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:13.738 INFO:tasks.rados.rados.0.smithi190.stdout:10: finishing write tid 3 to smithi19039307-10 2023-11-01T04:16:13.738 INFO:tasks.rados.rados.0.smithi190.stdout:10: oid 10 updating version 0 to 1 2023-11-01T04:16:13.738 INFO:tasks.rados.rados.0.smithi190.stdout:10: oid 10 updating version 1 to 2 2023-11-01T04:16:13.739 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 10 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2023-11-01T04:16:13.739 INFO:tasks.rados.rados.0.smithi190.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2023-11-01T04:16:13.739 INFO:tasks.rados.rados.0.smithi190.stdout:10: done (15 left) 2023-11-01T04:16:13.739 INFO:tasks.rados.rados.0.smithi190.stdout:24: write initial oid 24 2023-11-01T04:16:13.740 INFO:tasks.rados.rados.0.smithi190.stdout:24: seq_num 23 ranges {0=393216} 2023-11-01T04:16:13.741 INFO:tasks.rados.rados.0.smithi190.stdout:24: writing smithi19039307-24 from 0 to 393216 tid 1 2023-11-01T04:16:13.741 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:13.762 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:13.762 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:13.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-01T04:16:13.779 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:13.774+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:13.818 INFO:tasks.rados.rados.0.smithi190.stdout:9: finishing write tid 3 to smithi19039307-9 2023-11-01T04:16:13.819 INFO:tasks.rados.rados.0.smithi190.stdout:9: oid 9 updating version 0 to 1 2023-11-01T04:16:13.819 INFO:tasks.rados.rados.0.smithi190.stdout:9: oid 9 updating version 1 to 2 2023-11-01T04:16:13.819 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 9 v 2 (ObjNum 8 snap 0 seq_num 8) dirty exists 2023-11-01T04:16:13.819 INFO:tasks.rados.rados.0.smithi190.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2023-11-01T04:16:13.820 INFO:tasks.rados.rados.0.smithi190.stdout:9: done (15 left) 2023-11-01T04:16:13.820 INFO:tasks.rados.rados.0.smithi190.stdout:25: write initial oid 25 2023-11-01T04:16:13.820 INFO:tasks.rados.rados.0.smithi190.stdout:25: seq_num 24 ranges {0=409600} 2023-11-01T04:16:13.821 INFO:tasks.rados.rados.0.smithi190.stdout:25: writing smithi19039307-25 from 0 to 409600 tid 1 2023-11-01T04:16:13.821 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:13.821 INFO:tasks.rados.rados.0.smithi190.stdout:2: finishing write tid 3 to smithi19039307-2 2023-11-01T04:16:13.822 INFO:tasks.rados.rados.0.smithi190.stdout:2: oid 2 updating version 0 to 1 2023-11-01T04:16:13.822 INFO:tasks.rados.rados.0.smithi190.stdout:2: oid 2 updating version 1 to 2 2023-11-01T04:16:13.822 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2023-11-01T04:16:13.822 INFO:tasks.rados.rados.0.smithi190.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2023-11-01T04:16:13.823 INFO:tasks.rados.rados.0.smithi190.stdout:16: finishing write tid 3 to smithi19039307-16 2023-11-01T04:16:13.823 INFO:tasks.rados.rados.0.smithi190.stdout:16: oid 16 updating version 0 to 3 2023-11-01T04:16:13.823 INFO:tasks.rados.rados.0.smithi190.stdout:16: oid 16 updating version 3 to 4 2023-11-01T04:16:13.823 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 16 v 4 (ObjNum 15 snap 0 seq_num 15) dirty exists 2023-11-01T04:16:13.823 INFO:tasks.rados.rados.0.smithi190.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2023-11-01T04:16:13.824 INFO:tasks.rados.rados.0.smithi190.stdout:2: done (15 left) 2023-11-01T04:16:13.824 INFO:tasks.rados.rados.0.smithi190.stdout:16: done (14 left) 2023-11-01T04:16:13.824 INFO:tasks.rados.rados.0.smithi190.stdout:26: write initial oid 26 2023-11-01T04:16:13.824 INFO:tasks.rados.rados.0.smithi190.stdout:26: seq_num 25 ranges {0=425984} 2023-11-01T04:16:13.825 INFO:tasks.rados.rados.0.smithi190.stdout:26: writing smithi19039307-26 from 0 to 425984 tid 1 2023-11-01T04:16:13.825 INFO:tasks.rados.rados.0.smithi190.stdout:27: write initial oid 27 2023-11-01T04:16:13.825 INFO:tasks.rados.rados.0.smithi190.stdout:27: seq_num 26 ranges {0=442368} 2023-11-01T04:16:13.829 INFO:tasks.rados.rados.0.smithi190.stdout:27: writing smithi19039307-27 from 0 to 442368 tid 1 2023-11-01T04:16:13.829 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:13.830 INFO:tasks.rados.rados.0.smithi190.stdout:14: finishing write tid 1 to smithi19039307-14 2023-11-01T04:16:13.880 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:13.879+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:13.903 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:13.904 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:13.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-01T04:16:13.919 INFO:tasks.rados.rados.0.smithi190.stdout:12: finishing write tid 2 to smithi19039307-12 2023-11-01T04:16:13.922 INFO:teuthology.orchestra.run.smithi153.stderr:noscrub is unset 2023-11-01T04:16:13.923 INFO:teuthology.orchestra.run.smithi153.stderr:set 2.4 pg_upmap mapping to [9,8] 2023-11-01T04:16:13.934 INFO:tasks.rados.rados.0.smithi190.stdout:17: finishing write tid 1 to smithi19039307-17 2023-11-01T04:16:13.935 INFO:tasks.rados.rados.0.smithi190.stdout:17: finishing write tid 2 to smithi19039307-17 2023-11-01T04:16:13.938 INFO:tasks.rados.rados.0.smithi190.stdout:15: finishing write tid 3 to smithi19039307-15 2023-11-01T04:16:13.938 INFO:tasks.rados.rados.0.smithi190.stdout:15: oid 15 updating version 0 to 3 2023-11-01T04:16:13.938 INFO:tasks.rados.rados.0.smithi190.stdout:15: oid 15 updating version 3 to 4 2023-11-01T04:16:13.939 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 15 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2023-11-01T04:16:13.939 INFO:tasks.rados.rados.0.smithi190.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2023-11-01T04:16:13.939 INFO:tasks.rados.rados.0.smithi190.stdout:15: done (15 left) 2023-11-01T04:16:13.939 INFO:tasks.rados.rados.0.smithi190.stdout:28: write initial oid 28 2023-11-01T04:16:13.939 INFO:tasks.rados.rados.0.smithi190.stdout:28: seq_num 27 ranges {0=458752} 2023-11-01T04:16:13.941 INFO:tasks.rados.rados.0.smithi190.stdout:28: writing smithi19039307-28 from 0 to 458752 tid 1 2023-11-01T04:16:13.942 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:13.947 INFO:tasks.rados.rados.0.smithi190.stdout:19: finishing write tid 1 to smithi19039307-19 2023-11-01T04:16:13.947 INFO:tasks.rados.rados.0.smithi190.stdout:22: finishing write tid 1 to smithi19039307-22 2023-11-01T04:16:13.991 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:13.989+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:14.023 INFO:tasks.rados.rados.0.smithi190.stdout:11: finishing write tid 1 to smithi19039307-11 2023-11-01T04:16:14.023 INFO:tasks.rados.rados.0.smithi190.stdout:11: finishing write tid 2 to smithi19039307-11 2023-11-01T04:16:14.023 INFO:tasks.rados.rados.0.smithi190.stdout:5: finishing write tid 3 to smithi19039307-5 2023-11-01T04:16:14.024 INFO:tasks.rados.rados.0.smithi190.stdout:5: oid 5 updating version 0 to 1 2023-11-01T04:16:14.024 INFO:tasks.rados.rados.0.smithi190.stdout:5: oid 5 updating version 1 to 2 2023-11-01T04:16:14.024 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 5 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2023-11-01T04:16:14.024 INFO:tasks.rados.rados.0.smithi190.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2023-11-01T04:16:14.024 INFO:tasks.rados.rados.0.smithi190.stdout:5: done (15 left) 2023-11-01T04:16:14.025 INFO:tasks.rados.rados.0.smithi190.stdout:29: write initial oid 29 2023-11-01T04:16:14.025 INFO:tasks.rados.rados.0.smithi190.stdout:29: seq_num 28 ranges {0=475136} 2023-11-01T04:16:14.032 INFO:tasks.rados.rados.0.smithi190.stdout:29: writing smithi19039307-29 from 0 to 475136 tid 1 2023-11-01T04:16:14.033 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:14.033 INFO:tasks.rados.rados.0.smithi190.stdout:12: finishing write tid 3 to smithi19039307-12 2023-11-01T04:16:14.033 INFO:tasks.rados.rados.0.smithi190.stdout:12: oid 12 updating version 0 to 5 2023-11-01T04:16:14.033 INFO:tasks.rados.rados.0.smithi190.stdout:12: oid 12 updating version 5 to 6 2023-11-01T04:16:14.034 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 12 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2023-11-01T04:16:14.034 INFO:tasks.rados.rados.0.smithi190.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2023-11-01T04:16:14.034 INFO:tasks.rados.rados.0.smithi190.stdout:14: finishing write tid 2 to smithi19039307-14 2023-11-01T04:16:14.034 INFO:tasks.rados.rados.0.smithi190.stdout:18: finishing write tid 1 to smithi19039307-18 2023-11-01T04:16:14.034 INFO:tasks.rados.rados.0.smithi190.stdout:12: done (15 left) 2023-11-01T04:16:14.035 INFO:tasks.rados.rados.0.smithi190.stdout:30: write initial oid 30 2023-11-01T04:16:14.035 INFO:tasks.rados.rados.0.smithi190.stdout:30: seq_num 29 ranges {0=491520} 2023-11-01T04:16:14.041 INFO:tasks.rados.rados.0.smithi190.stdout:30: writing smithi19039307-30 from 0 to 491520 tid 1 2023-11-01T04:16:14.042 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:14.042 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:14.042 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:14.048 INFO:tasks.rados.rados.0.smithi190.stdout:19: finishing write tid 2 to smithi19039307-19 2023-11-01T04:16:14.052 INFO:tasks.rados.rados.0.smithi190.stdout:21: finishing write tid 1 to smithi19039307-21 2023-11-01T04:16:14.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-01T04:16:14.082 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:14.077+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:14.134 INFO:tasks.rados.rados.0.smithi190.stdout:18: finishing write tid 2 to smithi19039307-18 2023-11-01T04:16:14.134 INFO:tasks.rados.rados.0.smithi190.stdout:20: finishing write tid 1 to smithi19039307-20 2023-11-01T04:16:14.134 INFO:tasks.rados.rados.0.smithi190.stdout:20: finishing write tid 2 to smithi19039307-20 2023-11-01T04:16:14.141 INFO:tasks.rados.rados.0.smithi190.stdout:17: finishing write tid 3 to smithi19039307-17 2023-11-01T04:16:14.142 INFO:tasks.rados.rados.0.smithi190.stdout:17: oid 17 updating version 0 to 2 2023-11-01T04:16:14.142 INFO:tasks.rados.rados.0.smithi190.stdout:17: oid 17 version 2 is already newer than 1 2023-11-01T04:16:14.142 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 17 v 2 (ObjNum 16 snap 0 seq_num 16) dirty exists 2023-11-01T04:16:14.142 INFO:tasks.rados.rados.0.smithi190.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2023-11-01T04:16:14.142 INFO:tasks.rados.rados.0.smithi190.stdout:17: done (15 left) 2023-11-01T04:16:14.143 INFO:tasks.rados.rados.0.smithi190.stdout:31: write initial oid 31 2023-11-01T04:16:14.143 INFO:tasks.rados.rados.0.smithi190.stdout:31: seq_num 30 ranges {0=507904} 2023-11-01T04:16:14.146 INFO:tasks.rados.rados.0.smithi190.stdout:31: writing smithi19039307-31 from 0 to 507904 tid 1 2023-11-01T04:16:14.146 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:14.152 INFO:tasks.rados.rados.0.smithi190.stdout:22: finishing write tid 2 to smithi19039307-22 2023-11-01T04:16:14.183 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:14.180+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:14.226 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:14.227 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:14.234 INFO:tasks.rados.rados.0.smithi190.stdout:21: finishing write tid 2 to smithi19039307-21 2023-11-01T04:16:14.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-01T04:16:14.245 INFO:tasks.rados.rados.0.smithi190.stdout:23: finishing write tid 1 to smithi19039307-23 2023-11-01T04:16:14.283 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:14.281+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:14.336 INFO:tasks.rados.rados.0.smithi190.stdout:18: finishing write tid 3 to smithi19039307-18 2023-11-01T04:16:14.337 INFO:tasks.rados.rados.0.smithi190.stdout:18: oid 18 updating version 0 to 5 2023-11-01T04:16:14.337 INFO:tasks.rados.rados.0.smithi190.stdout:18: oid 18 updating version 5 to 6 2023-11-01T04:16:14.337 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-01T04:16:14.337 INFO:tasks.rados.rados.0.smithi190.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2023-11-01T04:16:14.338 INFO:tasks.rados.rados.0.smithi190.stdout:18: done (15 left) 2023-11-01T04:16:14.338 INFO:tasks.rados.rados.0.smithi190.stdout:32: write initial oid 32 2023-11-01T04:16:14.338 INFO:tasks.rados.rados.0.smithi190.stdout:32: seq_num 31 ranges {0=524288} 2023-11-01T04:16:14.345 INFO:tasks.rados.rados.0.smithi190.stdout:32: writing smithi19039307-32 from 0 to 524288 tid 1 2023-11-01T04:16:14.345 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:14.354 INFO:tasks.rados.rados.0.smithi190.stdout:31: finishing write tid 1 to smithi19039307-31 2023-11-01T04:16:14.355 INFO:tasks.rados.rados.0.smithi190.stdout:14: finishing write tid 3 to smithi19039307-14 2023-11-01T04:16:14.355 INFO:tasks.rados.rados.0.smithi190.stdout:14: oid 14 updating version 0 to 3 2023-11-01T04:16:14.355 INFO:tasks.rados.rados.0.smithi190.stdout:14: oid 14 updating version 3 to 4 2023-11-01T04:16:14.355 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 14 v 4 (ObjNum 13 snap 0 seq_num 13) dirty exists 2023-11-01T04:16:14.355 INFO:tasks.rados.rados.0.smithi190.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2023-11-01T04:16:14.356 INFO:tasks.rados.rados.0.smithi190.stdout:14: done (15 left) 2023-11-01T04:16:14.356 INFO:tasks.rados.rados.0.smithi190.stdout:33: write initial oid 33 2023-11-01T04:16:14.356 INFO:tasks.rados.rados.0.smithi190.stdout:33: seq_num 32 ranges {0=540672} 2023-11-01T04:16:14.363 INFO:tasks.rados.rados.0.smithi190.stdout:33: writing smithi19039307-33 from 0 to 540672 tid 1 2023-11-01T04:16:14.364 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:14.364 INFO:tasks.rados.rados.0.smithi190.stdout:31: finishing write tid 2 to smithi19039307-31 2023-11-01T04:16:14.364 INFO:tasks.rados.rados.0.smithi190.stdout:26: finishing write tid 1 to smithi19039307-26 2023-11-01T04:16:14.365 INFO:tasks.rados.rados.0.smithi190.stdout:26: finishing write tid 2 to smithi19039307-26 2023-11-01T04:16:14.368 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:14.368 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:14.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-01T04:16:14.384 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:14.381+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:14.426 INFO:tasks.rados.rados.0.smithi190.stdout:11: finishing write tid 3 to smithi19039307-11 2023-11-01T04:16:14.426 INFO:tasks.rados.rados.0.smithi190.stdout:11: oid 11 updating version 0 to 4 2023-11-01T04:16:14.427 INFO:tasks.rados.rados.0.smithi190.stdout:11: oid 11 version 4 is already newer than 3 2023-11-01T04:16:14.427 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 11 v 4 (ObjNum 10 snap 0 seq_num 10) dirty exists 2023-11-01T04:16:14.427 INFO:tasks.rados.rados.0.smithi190.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2023-11-01T04:16:14.427 INFO:tasks.rados.rados.0.smithi190.stdout:25: finishing write tid 1 to smithi19039307-25 2023-11-01T04:16:14.427 INFO:tasks.rados.rados.0.smithi190.stdout:11: done (15 left) 2023-11-01T04:16:14.427 INFO:tasks.rados.rados.0.smithi190.stdout:34: write initial oid 34 2023-11-01T04:16:14.428 INFO:tasks.rados.rados.0.smithi190.stdout:34: seq_num 33 ranges {0=557056} 2023-11-01T04:16:14.435 INFO:tasks.rados.rados.0.smithi190.stdout:34: writing smithi19039307-34 from 0 to 557056 tid 1 2023-11-01T04:16:14.435 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:14.437 INFO:tasks.rados.rados.0.smithi190.stdout:21: finishing write tid 3 to smithi19039307-21 2023-11-01T04:16:14.438 INFO:tasks.rados.rados.0.smithi190.stdout:21: oid 21 updating version 0 to 1 2023-11-01T04:16:14.438 INFO:tasks.rados.rados.0.smithi190.stdout:21: oid 21 updating version 1 to 2 2023-11-01T04:16:14.438 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 21 v 2 (ObjNum 20 snap 0 seq_num 20) dirty exists 2023-11-01T04:16:14.438 INFO:tasks.rados.rados.0.smithi190.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2023-11-01T04:16:14.438 INFO:tasks.rados.rados.0.smithi190.stdout:21: done (15 left) 2023-11-01T04:16:14.439 INFO:tasks.rados.rados.0.smithi190.stdout:35: write initial oid 35 2023-11-01T04:16:14.439 INFO:tasks.rados.rados.0.smithi190.stdout:35: seq_num 34 ranges {0=573440} 2023-11-01T04:16:14.448 INFO:tasks.rados.rados.0.smithi190.stdout:35: writing smithi19039307-35 from 0 to 573440 tid 1 2023-11-01T04:16:14.448 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:14.449 INFO:tasks.rados.rados.0.smithi190.stdout:30: finishing write tid 1 to smithi19039307-30 2023-11-01T04:16:14.454 INFO:tasks.rados.rados.0.smithi190.stdout:19: finishing write tid 3 to smithi19039307-19 2023-11-01T04:16:14.455 INFO:tasks.rados.rados.0.smithi190.stdout:19: oid 19 updating version 0 to 1 2023-11-01T04:16:14.455 INFO:tasks.rados.rados.0.smithi190.stdout:19: oid 19 updating version 1 to 2 2023-11-01T04:16:14.455 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 19 v 2 (ObjNum 18 snap 0 seq_num 18) dirty exists 2023-11-01T04:16:14.455 INFO:tasks.rados.rados.0.smithi190.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2023-11-01T04:16:14.455 INFO:tasks.rados.rados.0.smithi190.stdout:23: finishing write tid 2 to smithi19039307-23 2023-11-01T04:16:14.456 INFO:tasks.rados.rados.0.smithi190.stdout:19: done (15 left) 2023-11-01T04:16:14.456 INFO:tasks.rados.rados.0.smithi190.stdout:36: write initial oid 36 2023-11-01T04:16:14.456 INFO:tasks.rados.rados.0.smithi190.stdout:36: seq_num 35 ranges {0=589824} 2023-11-01T04:16:14.464 INFO:tasks.rados.rados.0.smithi190.stdout:36: writing smithi19039307-36 from 0 to 589824 tid 1 2023-11-01T04:16:14.464 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:14.485 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:14.482+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:14.506 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:14.506 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:14.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-01T04:16:14.526 INFO:tasks.rados.rados.0.smithi190.stdout:25: finishing write tid 2 to smithi19039307-25 2023-11-01T04:16:14.538 INFO:tasks.rados.rados.0.smithi190.stdout:30: finishing write tid 2 to smithi19039307-30 2023-11-01T04:16:14.552 INFO:tasks.rados.rados.0.smithi190.stdout:27: finishing write tid 1 to smithi19039307-27 2023-11-01T04:16:14.553 INFO:tasks.rados.rados.0.smithi190.stdout:22: finishing write tid 3 to smithi19039307-22 2023-11-01T04:16:14.553 INFO:tasks.rados.rados.0.smithi190.stdout:22: oid 22 updating version 0 to 3 2023-11-01T04:16:14.554 INFO:tasks.rados.rados.0.smithi190.stdout:22: oid 22 updating version 3 to 4 2023-11-01T04:16:14.554 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 22 v 4 (ObjNum 21 snap 0 seq_num 21) dirty exists 2023-11-01T04:16:14.554 INFO:tasks.rados.rados.0.smithi190.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2023-11-01T04:16:14.554 INFO:tasks.rados.rados.0.smithi190.stdout:22: done (15 left) 2023-11-01T04:16:14.554 INFO:tasks.rados.rados.0.smithi190.stdout:37: write initial oid 37 2023-11-01T04:16:14.555 INFO:tasks.rados.rados.0.smithi190.stdout:37: seq_num 36 ranges {0=606208} 2023-11-01T04:16:14.558 INFO:tasks.rados.rados.0.smithi190.stdout:37: writing smithi19039307-37 from 0 to 606208 tid 1 2023-11-01T04:16:14.559 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:14.559 INFO:tasks.rados.rados.0.smithi190.stdout:26: finishing write tid 3 to smithi19039307-26 2023-11-01T04:16:14.559 INFO:tasks.rados.rados.0.smithi190.stdout:26: oid 26 updating version 0 to 3 2023-11-01T04:16:14.559 INFO:tasks.rados.rados.0.smithi190.stdout:26: oid 26 updating version 3 to 4 2023-11-01T04:16:14.559 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 26 v 4 (ObjNum 25 snap 0 seq_num 25) dirty exists 2023-11-01T04:16:14.560 INFO:tasks.rados.rados.0.smithi190.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2023-11-01T04:16:14.560 INFO:tasks.rados.rados.0.smithi190.stdout:26: done (15 left) 2023-11-01T04:16:14.560 INFO:tasks.rados.rados.0.smithi190.stdout:38: write initial oid 38 2023-11-01T04:16:14.560 INFO:tasks.rados.rados.0.smithi190.stdout:38: seq_num 37 ranges {0=622592} 2023-11-01T04:16:14.563 INFO:tasks.rados.rados.0.smithi190.stdout:38: writing smithi19039307-38 from 0 to 622592 tid 1 2023-11-01T04:16:14.564 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:14.564 INFO:tasks.rados.rados.0.smithi190.stdout:27: finishing write tid 2 to smithi19039307-27 2023-11-01T04:16:14.585 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:14.583+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:14.653 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:14.653 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:14.660 INFO:tasks.rados.rados.0.smithi190.stdout:31: finishing write tid 3 to smithi19039307-31 2023-11-01T04:16:14.660 INFO:tasks.rados.rados.0.smithi190.stdout:31: oid 31 updating version 0 to 7 2023-11-01T04:16:14.660 INFO:tasks.rados.rados.0.smithi190.stdout:31: oid 31 updating version 7 to 8 2023-11-01T04:16:14.660 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 31 v 8 (ObjNum 30 snap 0 seq_num 30) dirty exists 2023-11-01T04:16:14.660 INFO:tasks.rados.rados.0.smithi190.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2023-11-01T04:16:14.661 INFO:tasks.rados.rados.0.smithi190.stdout:31: done (15 left) 2023-11-01T04:16:14.661 INFO:tasks.rados.rados.0.smithi190.stdout:39: write initial oid 39 2023-11-01T04:16:14.661 INFO:tasks.rados.rados.0.smithi190.stdout:39: seq_num 38 ranges {0=638976} 2023-11-01T04:16:14.669 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-01T04:16:14.671 INFO:tasks.rados.rados.0.smithi190.stdout:39: writing smithi19039307-39 from 0 to 638976 tid 1 2023-11-01T04:16:14.671 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:14.686 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:14.684+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:14.742 INFO:tasks.rados.rados.0.smithi190.stdout:30: finishing write tid 3 to smithi19039307-30 2023-11-01T04:16:14.743 INFO:tasks.rados.rados.0.smithi190.stdout:30: oid 30 updating version 0 to 4 2023-11-01T04:16:14.743 INFO:tasks.rados.rados.0.smithi190.stdout:30: oid 30 version 4 is already newer than 3 2023-11-01T04:16:14.743 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 30 v 4 (ObjNum 29 snap 0 seq_num 29) dirty exists 2023-11-01T04:16:14.743 INFO:tasks.rados.rados.0.smithi190.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2023-11-01T04:16:14.743 INFO:tasks.rados.rados.0.smithi190.stdout:30: done (15 left) 2023-11-01T04:16:14.744 INFO:tasks.rados.rados.0.smithi190.stdout:40: write initial oid 40 2023-11-01T04:16:14.744 INFO:tasks.rados.rados.0.smithi190.stdout:40: seq_num 39 ranges {0=671744} 2023-11-01T04:16:14.754 INFO:tasks.rados.rados.0.smithi190.stdout:40: writing smithi19039307-40 from 0 to 671744 tid 1 2023-11-01T04:16:14.754 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:14.754 INFO:tasks.rados.rados.0.smithi190.stdout:32: finishing write tid 1 to smithi19039307-32 2023-11-01T04:16:14.755 INFO:tasks.rados.rados.0.smithi190.stdout:32: finishing write tid 2 to smithi19039307-32 2023-11-01T04:16:14.760 INFO:tasks.rados.rados.0.smithi190.stdout:28: finishing write tid 1 to smithi19039307-28 2023-11-01T04:16:14.760 INFO:tasks.rados.rados.0.smithi190.stdout:28: finishing write tid 2 to smithi19039307-28 2023-11-01T04:16:14.761 INFO:tasks.rados.rados.0.smithi190.stdout:35: finishing write tid 1 to smithi19039307-35 2023-11-01T04:16:14.780 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-01T04:16:14.787 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:14.784+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:14.801 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:14.801 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:14.816 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-01T04:16:14.844 INFO:tasks.rados.rados.0.smithi190.stdout:34: finishing write tid 1 to smithi19039307-34 2023-11-01T04:16:14.844 INFO:tasks.rados.rados.0.smithi190.stdout:34: finishing write tid 2 to smithi19039307-34 2023-11-01T04:16:14.845 INFO:tasks.rados.rados.0.smithi190.stdout:20: finishing write tid 3 to smithi19039307-20 2023-11-01T04:16:14.845 INFO:tasks.rados.rados.0.smithi190.stdout:20: oid 20 updating version 0 to 7 2023-11-01T04:16:14.845 INFO:tasks.rados.rados.0.smithi190.stdout:20: oid 20 updating version 7 to 8 2023-11-01T04:16:14.846 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 20 v 8 (ObjNum 19 snap 0 seq_num 19) dirty exists 2023-11-01T04:16:14.846 INFO:tasks.rados.rados.0.smithi190.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2023-11-01T04:16:14.846 INFO:tasks.rados.rados.0.smithi190.stdout:20: done (15 left) 2023-11-01T04:16:14.846 INFO:tasks.rados.rados.0.smithi190.stdout:41: write initial oid 41 2023-11-01T04:16:14.846 INFO:tasks.rados.rados.0.smithi190.stdout:41: seq_num 40 ranges {0=688128} 2023-11-01T04:16:14.857 INFO:tasks.rados.rados.0.smithi190.stdout:41: writing smithi19039307-41 from 0 to 688128 tid 1 2023-11-01T04:16:14.857 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:14.857 INFO:tasks.rados.rados.0.smithi190.stdout:25: finishing write tid 3 to smithi19039307-25 2023-11-01T04:16:14.858 INFO:tasks.rados.rados.0.smithi190.stdout:25: oid 25 updating version 0 to 5 2023-11-01T04:16:14.858 INFO:tasks.rados.rados.0.smithi190.stdout:25: oid 25 updating version 5 to 6 2023-11-01T04:16:14.858 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2023-11-01T04:16:14.858 INFO:tasks.rados.rados.0.smithi190.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2023-11-01T04:16:14.858 INFO:tasks.rados.rados.0.smithi190.stdout:25: done (15 left) 2023-11-01T04:16:14.859 INFO:tasks.rados.rados.0.smithi190.stdout:42: write initial oid 42 2023-11-01T04:16:14.859 INFO:tasks.rados.rados.0.smithi190.stdout:42: seq_num 41 ranges {0=704512} 2023-11-01T04:16:14.871 INFO:tasks.rados.rados.0.smithi190.stdout:42: writing smithi19039307-42 from 0 to 704512 tid 1 2023-11-01T04:16:14.871 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:14.872 INFO:tasks.rados.rados.0.smithi190.stdout:32: finishing write tid 3 to smithi19039307-32 2023-11-01T04:16:14.872 INFO:tasks.rados.rados.0.smithi190.stdout:32: oid 32 updating version 0 to 1 2023-11-01T04:16:14.872 INFO:tasks.rados.rados.0.smithi190.stdout:32: oid 32 updating version 1 to 2 2023-11-01T04:16:14.872 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 32 v 2 (ObjNum 31 snap 0 seq_num 31) dirty exists 2023-11-01T04:16:14.872 INFO:tasks.rados.rados.0.smithi190.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2023-11-01T04:16:14.873 INFO:tasks.rados.rados.0.smithi190.stdout:35: finishing write tid 2 to smithi19039307-35 2023-11-01T04:16:14.873 INFO:tasks.rados.rados.0.smithi190.stdout:37: finishing write tid 1 to smithi19039307-37 2023-11-01T04:16:14.873 INFO:tasks.rados.rados.0.smithi190.stdout:24: finishing write tid 1 to smithi19039307-24 2023-11-01T04:16:14.873 INFO:tasks.rados.rados.0.smithi190.stdout:24: finishing write tid 2 to smithi19039307-24 2023-11-01T04:16:14.873 INFO:tasks.rados.rados.0.smithi190.stdout:29: finishing write tid 1 to smithi19039307-29 2023-11-01T04:16:14.873 INFO:tasks.rados.rados.0.smithi190.stdout:29: finishing write tid 2 to smithi19039307-29 2023-11-01T04:16:14.874 INFO:tasks.rados.rados.0.smithi190.stdout:32: done (15 left) 2023-11-01T04:16:14.874 INFO:tasks.rados.rados.0.smithi190.stdout:43: write initial oid 43 2023-11-01T04:16:14.874 INFO:tasks.rados.rados.0.smithi190.stdout:43: seq_num 42 ranges {0=720896} 2023-11-01T04:16:14.884 INFO:tasks.rados.rados.0.smithi190.stdout:43: writing smithi19039307-43 from 0 to 720896 tid 1 2023-11-01T04:16:14.884 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:14.887 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:14.888+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:14.938 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:14.938 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-11-01T04:16:14.943 INFO:tasks.rados.rados.0.smithi190.stdout:36: finishing write tid 1 to smithi19039307-36 2023-11-01T04:16:14.944 INFO:tasks.rados.rados.0.smithi190.stdout:36: finishing write tid 2 to smithi19039307-36 2023-11-01T04:16:14.959 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:14.959 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-01T04:16:14.969 INFO:tasks.rados.rados.0.smithi190.stdout:37: finishing write tid 2 to smithi19039307-37 2023-11-01T04:16:14.974 INFO:tasks.rados.rados.0.smithi190.stdout:39: finishing write tid 1 to smithi19039307-39 2023-11-01T04:16:14.980 INFO:tasks.rados.rados.0.smithi190.stdout:42: finishing write tid 1 to smithi19039307-42 2023-11-01T04:16:14.982 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:14.982 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:16:14.983 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2023-11-01T04:16:14.983 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2023-11-01T04:16:14.983 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:14.988 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:14.987+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:14.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-01T04:16:15.062 INFO:tasks.rados.rados.0.smithi190.stdout:23: finishing write tid 3 to smithi19039307-23 2023-11-01T04:16:15.062 INFO:tasks.rados.rados.0.smithi190.stdout:23: oid 23 updating version 0 to 5 2023-11-01T04:16:15.063 INFO:tasks.rados.rados.0.smithi190.stdout:23: oid 23 updating version 5 to 6 2023-11-01T04:16:15.063 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2023-11-01T04:16:15.063 INFO:tasks.rados.rados.0.smithi190.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2023-11-01T04:16:15.063 INFO:tasks.rados.rados.0.smithi190.stdout:23: done (15 left) 2023-11-01T04:16:15.063 INFO:tasks.rados.rados.0.smithi190.stdout:44: write initial oid 44 2023-11-01T04:16:15.063 INFO:tasks.rados.rados.0.smithi190.stdout:44: seq_num 43 ranges {0=737280} 2023-11-01T04:16:15.070 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:15.071 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-01T04:16:15.076 INFO:tasks.rados.rados.0.smithi190.stdout:44: writing smithi19039307-44 from 0 to 737280 tid 1 2023-11-01T04:16:15.076 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:15.076 INFO:tasks.rados.rados.0.smithi190.stdout:34: finishing write tid 3 to smithi19039307-34 2023-11-01T04:16:15.076 INFO:tasks.rados.rados.0.smithi190.stdout:34: oid 34 updating version 0 to 2 2023-11-01T04:16:15.077 INFO:tasks.rados.rados.0.smithi190.stdout:34: oid 34 version 2 is already newer than 1 2023-11-01T04:16:15.077 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 34 v 2 (ObjNum 33 snap 0 seq_num 33) dirty exists 2023-11-01T04:16:15.077 INFO:tasks.rados.rados.0.smithi190.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2023-11-01T04:16:15.077 INFO:tasks.rados.rados.0.smithi190.stdout:40: finishing write tid 1 to smithi19039307-40 2023-11-01T04:16:15.077 INFO:tasks.rados.rados.0.smithi190.stdout:38: finishing write tid 1 to smithi19039307-38 2023-11-01T04:16:15.078 INFO:tasks.rados.rados.0.smithi190.stdout:38: finishing write tid 2 to smithi19039307-38 2023-11-01T04:16:15.078 INFO:tasks.rados.rados.0.smithi190.stdout:39: finishing write tid 2 to smithi19039307-39 2023-11-01T04:16:15.078 INFO:tasks.rados.rados.0.smithi190.stdout:40: finishing write tid 2 to smithi19039307-40 2023-11-01T04:16:15.078 INFO:tasks.rados.rados.0.smithi190.stdout:34: done (15 left) 2023-11-01T04:16:15.079 INFO:tasks.rados.rados.0.smithi190.stdout:45: write initial oid 45 2023-11-01T04:16:15.079 INFO:tasks.rados.rados.0.smithi190.stdout:45: seq_num 44 ranges {0=753664} 2023-11-01T04:16:15.088 INFO:tasks.rados.rados.0.smithi190.stdout:45: writing smithi19039307-45 from 0 to 753664 tid 1 2023-11-01T04:16:15.089 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:15.089 INFO:tasks.rados.rados.0.smithi190.stdout:41: finishing write tid 1 to smithi19039307-41 2023-11-01T04:16:15.089 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:15.087+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:15.090 INFO:tasks.rados.rados.0.smithi190.stdout:33: finishing write tid 1 to smithi19039307-33 2023-11-01T04:16:15.092 INFO:tasks.rados.rados.0.smithi190.stdout:33: finishing write tid 2 to smithi19039307-33 2023-11-01T04:16:15.092 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:15.093 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-01T04:16:15.093 INFO:tasks.rados.rados.0.smithi190.stdout:43: finishing write tid 1 to smithi19039307-43 2023-11-01T04:16:15.094 INFO:tasks.rados.rados.0.smithi190.stdout:43: finishing write tid 2 to smithi19039307-43 2023-11-01T04:16:15.108 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:15.109 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-01T04:16:15.114 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:15.114 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2023-11-01T04:16:15.114 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2023-11-01T04:16:15.114 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:16:15.114 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.115 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 35/31 ECSubWriteReply(tid=6, last_complete=35'2, committed=1, applied=1))", 2023-11-01T04:16:15.115 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:14.946969+0000", 2023-11-01T04:16:15.115 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.16764321200000001, 2023-11-01T04:16:15.115 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.017084144999999998, 2023-11-01T04:16:15.115 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.116 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.116 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.116 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.116 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.116 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.946969+0000", 2023-11-01T04:16:15.117 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.117 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.117 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.117 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.946969+0000", 2023-11-01T04:16:15.117 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.118 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.118 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.118 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.946970+0000", 2023-11-01T04:16:15.118 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.161e-06 2023-11-01T04:16:15.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.119 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.119 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.946977+0000", 2023-11-01T04:16:15.119 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4590000000000001e-06 2023-11-01T04:16:15.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.120 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.120 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.120 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.946990+0000", 2023-11-01T04:16:15.120 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2883e-05 2023-11-01T04:16:15.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.121 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.121 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.121 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.947000+0000", 2023-11-01T04:16:15.121 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0224000000000001e-05 2023-11-01T04:16:15.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.122 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.122 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.964035+0000", 2023-11-01T04:16:15.122 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.017034443999999999 2023-11-01T04:16:15.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.123 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.123 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.964053+0000", 2023-11-01T04:16:15.123 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7974000000000001e-05 2023-11-01T04:16:15.123 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.123 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.124 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.124 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.124 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.fs0 35/31 ECSubReadReply(tid=7, attrs_read=0))", 2023-11-01T04:16:15.124 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:14.947055+0000", 2023-11-01T04:16:15.124 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.16755721000000001, 2023-11-01T04:16:15.125 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.022830284999999999, 2023-11-01T04:16:15.125 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.125 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.125 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.125 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.126 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.126 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.947055+0000", 2023-11-01T04:16:15.126 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.126 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.127 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.127 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.947055+0000", 2023-11-01T04:16:15.127 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.127 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.128 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.128 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.947055+0000", 2023-11-01T04:16:15.128 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9299999999999998e-07 2023-11-01T04:16:15.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.128 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.128 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.947066+0000", 2023-11-01T04:16:15.129 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0706999999999999e-05 2023-11-01T04:16:15.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.129 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.129 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.130 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.947073+0000", 2023-11-01T04:16:15.130 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6090000000000003e-06 2023-11-01T04:16:15.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.130 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.130 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.130 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.947079+0000", 2023-11-01T04:16:15.131 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.857e-06 2023-11-01T04:16:15.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.131 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.131 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.131 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.969851+0000", 2023-11-01T04:16:15.132 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.022772100999999999 2023-11-01T04:16:15.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.132 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.132 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.132 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.969885+0000", 2023-11-01T04:16:15.132 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.3518e-05 2023-11-01T04:16:15.133 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.133 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.133 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.133 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.133 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 35/31 ECSubWriteReply(tid=6, last_complete=35'2, committed=1, applied=1))", 2023-11-01T04:16:15.134 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:14.948272+0000", 2023-11-01T04:16:15.134 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.16633959400000001, 2023-11-01T04:16:15.134 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.015845185000000001, 2023-11-01T04:16:15.134 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.134 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.135 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.135 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.135 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.135 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.948272+0000", 2023-11-01T04:16:15.135 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.136 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.136 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.136 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.948272+0000", 2023-11-01T04:16:15.136 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.137 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.137 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.137 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.948273+0000", 2023-11-01T04:16:15.137 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.09e-07 2023-11-01T04:16:15.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.137 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.138 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.138 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.948278+0000", 2023-11-01T04:16:15.138 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.237e-06 2023-11-01T04:16:15.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.138 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.138 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.948290+0000", 2023-11-01T04:16:15.139 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1436e-05 2023-11-01T04:16:15.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.139 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.139 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.948297+0000", 2023-11-01T04:16:15.140 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.498e-06 2023-11-01T04:16:15.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.140 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.140 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.140 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.964100+0000", 2023-11-01T04:16:15.141 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.015802448 2023-11-01T04:16:15.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.141 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.141 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.141 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.964117+0000", 2023-11-01T04:16:15.141 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7757e-05 2023-11-01T04:16:15.142 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.142 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.142 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.143 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.fs0 35/31 ECSubReadReply(tid=7, attrs_read=0))", 2023-11-01T04:16:15.143 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:14.969731+0000", 2023-11-01T04:16:15.143 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.14488090100000001, 2023-11-01T04:16:15.143 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002855495, 2023-11-01T04:16:15.143 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.143 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.144 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.144 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.144 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.144 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.969731+0000", 2023-11-01T04:16:15.144 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.145 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.145 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.145 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.969731+0000", 2023-11-01T04:16:15.145 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.145 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.146 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.969732+0000", 2023-11-01T04:16:15.146 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.187e-06 2023-11-01T04:16:15.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.146 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.147 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.147 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.969744+0000", 2023-11-01T04:16:15.147 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2201999999999999e-05 2023-11-01T04:16:15.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.147 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.147 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.148 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.969757+0000", 2023-11-01T04:16:15.148 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2828e-05 2023-11-01T04:16:15.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.148 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.148 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.148 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.969764+0000", 2023-11-01T04:16:15.149 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1969999999999999e-06 2023-11-01T04:16:15.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.149 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.149 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.149 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972545+0000", 2023-11-01T04:16:15.150 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0027805410000000001 2023-11-01T04:16:15.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.150 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.150 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.150 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972586+0000", 2023-11-01T04:16:15.150 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1539999999999999e-05 2023-11-01T04:16:15.151 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.151 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.151 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.151 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.152 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 35/31 ECSubWriteReply(tid=6, last_complete=35'2, committed=1, applied=1))", 2023-11-01T04:16:15.152 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:14.972426+0000", 2023-11-01T04:16:15.152 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.14218620500000001, 2023-11-01T04:16:15.152 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00064765600000000003, 2023-11-01T04:16:15.152 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.152 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.153 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.153 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.153 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.153 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972426+0000", 2023-11-01T04:16:15.153 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.154 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.154 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.154 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972426+0000", 2023-11-01T04:16:15.154 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.155 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.155 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.155 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972427+0000", 2023-11-01T04:16:15.155 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7399999999999991e-07 2023-11-01T04:16:15.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.155 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.156 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.156 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972432+0000", 2023-11-01T04:16:15.156 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.429e-06 2023-11-01T04:16:15.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.156 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.157 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.157 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972446+0000", 2023-11-01T04:16:15.157 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.432e-05 2023-11-01T04:16:15.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.157 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.157 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.158 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972454+0000", 2023-11-01T04:16:15.158 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0190000000000006e-06 2023-11-01T04:16:15.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.158 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.158 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.158 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.973031+0000", 2023-11-01T04:16:15.159 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00057695000000000001 2023-11-01T04:16:15.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.159 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.159 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.159 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.973073+0000", 2023-11-01T04:16:15.159 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1964e-05 2023-11-01T04:16:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.160 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s2 35/31 ECSubWrite(tid=182, reqid=client.4559.0:86, at_version=0'0, trim_to=0'0, roll_forward_to=35'12))", 2023-11-01T04:16:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:14.972504+0000", 2023-11-01T04:16:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.142108082, 2023-11-01T04:16:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000862515, 2023-11-01T04:16:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.161 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.162 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.162 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.162 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.162 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972504+0000", 2023-11-01T04:16:15.162 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.163 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.163 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.163 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972504+0000", 2023-11-01T04:16:15.163 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.164 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.164 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.164 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972504+0000", 2023-11-01T04:16:15.164 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3300000000000002e-07 2023-11-01T04:16:15.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.164 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.165 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.165 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972520+0000", 2023-11-01T04:16:15.165 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5503e-05 2023-11-01T04:16:15.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.165 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.165 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.166 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972533+0000", 2023-11-01T04:16:15.166 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3543000000000001e-05 2023-11-01T04:16:15.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.166 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.166 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.167 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972543+0000", 2023-11-01T04:16:15.167 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.1959999999999997e-06 2023-11-01T04:16:15.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.167 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.167 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.167 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972627+0000", 2023-11-01T04:16:15.168 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4807000000000003e-05 2023-11-01T04:16:15.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.168 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.168 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.168 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972637+0000", 2023-11-01T04:16:15.168 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5650000000000007e-06 2023-11-01T04:16:15.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.169 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.169 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.169 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.973328+0000", 2023-11-01T04:16:15.169 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00069092999999999997 2023-11-01T04:16:15.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.170 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.170 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.170 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.973366+0000", 2023-11-01T04:16:15.171 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8438e-05 2023-11-01T04:16:15.171 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.171 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.171 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.171 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.172 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.2s2 35/31 ECSubRead(tid=183, to_read={3:4b7227f5:::smithi19039307-29:head=0,4096,0}, subchunks={3:4b7227f5:::smithi19039307-29:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:15.172 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:14.972567+0000", 2023-11-01T04:16:15.172 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.14204477099999999, 2023-11-01T04:16:15.172 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100601062, 2023-11-01T04:16:15.172 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.173 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.173 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.173 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.173 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.173 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972567+0000", 2023-11-01T04:16:15.173 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.174 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.174 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.174 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972567+0000", 2023-11-01T04:16:15.174 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.175 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.175 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.175 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972567+0000", 2023-11-01T04:16:15.175 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6399999999999998e-07 2023-11-01T04:16:15.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.175 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.176 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.176 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972584+0000", 2023-11-01T04:16:15.176 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7079e-05 2023-11-01T04:16:15.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.176 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.176 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.177 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972612+0000", 2023-11-01T04:16:15.177 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7951999999999999e-05 2023-11-01T04:16:15.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.177 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.177 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.177 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972632+0000", 2023-11-01T04:16:15.178 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9242999999999999e-05 2023-11-01T04:16:15.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.178 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.178 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.178 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.072922+0000", 2023-11-01T04:16:15.178 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.1002905 2023-11-01T04:16:15.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.179 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.179 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.179 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.073168+0000", 2023-11-01T04:16:15.179 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00024592400000000001 2023-11-01T04:16:15.179 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.180 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.180 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.180 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.180 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.fs0 35/31 ECSubReadReply(tid=7, attrs_read=0))", 2023-11-01T04:16:15.180 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:14.972671+0000", 2023-11-01T04:16:15.181 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.14194127400000001, 2023-11-01T04:16:15.181 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.091429504999999994, 2023-11-01T04:16:15.181 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.181 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.181 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.181 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.182 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.182 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972671+0000", 2023-11-01T04:16:15.182 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.182 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.182 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.183 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972671+0000", 2023-11-01T04:16:15.183 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.183 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.183 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.183 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972671+0000", 2023-11-01T04:16:15.184 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4300000000000003e-07 2023-11-01T04:16:15.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.184 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.184 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.184 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972691+0000", 2023-11-01T04:16:15.184 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0069999999999999e-05 2023-11-01T04:16:15.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.185 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.185 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.185 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972708+0000", 2023-11-01T04:16:15.185 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6795000000000001e-05 2023-11-01T04:16:15.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.186 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.186 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.186 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972721+0000", 2023-11-01T04:16:15.186 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3193999999999999e-05 2023-11-01T04:16:15.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.186 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.187 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.187 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.063917+0000", 2023-11-01T04:16:15.187 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.091196174000000005 2023-11-01T04:16:15.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.187 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.187 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.188 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.064100+0000", 2023-11-01T04:16:15.188 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00018272899999999999 2023-11-01T04:16:15.188 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.188 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.188 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.189 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.3s3 35/31 ECSubWrite(tid=184, reqid=client.4559.0:110, at_version=0'0, trim_to=0'0, roll_forward_to=35'4))", 2023-11-01T04:16:15.189 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:14.972750+0000", 2023-11-01T04:16:15.189 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.14186174300000001, 2023-11-01T04:16:15.189 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101005662, 2023-11-01T04:16:15.189 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.190 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.190 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.190 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.190 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.190 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972750+0000", 2023-11-01T04:16:15.190 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.191 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.191 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972750+0000", 2023-11-01T04:16:15.191 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.192 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.192 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972751+0000", 2023-11-01T04:16:15.192 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7599999999999997e-07 2023-11-01T04:16:15.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.193 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.193 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972768+0000", 2023-11-01T04:16:15.193 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7176e-05 2023-11-01T04:16:15.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.193 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.194 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972790+0000", 2023-11-01T04:16:15.194 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2453e-05 2023-11-01T04:16:15.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.194 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.194 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.194 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972807+0000", 2023-11-01T04:16:15.195 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6549e-05 2023-11-01T04:16:15.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.195 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.195 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.195 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.073187+0000", 2023-11-01T04:16:15.195 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100380128 2023-11-01T04:16:15.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.196 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.196 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.196 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.073192+0000", 2023-11-01T04:16:15.196 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9729999999999997e-06 2023-11-01T04:16:15.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.197 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.197 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.197 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.073732+0000", 2023-11-01T04:16:15.197 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00054037300000000003 2023-11-01T04:16:15.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.198 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.198 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.198 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.073756+0000", 2023-11-01T04:16:15.198 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3434e-05 2023-11-01T04:16:15.198 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.198 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.199 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.199 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.199 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.3s3 35/31 ECSubRead(tid=185, to_read={3:ca2966a4:::smithi19039307-37:head=0,4096,0}, subchunks={3:ca2966a4:::smithi19039307-37:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:15.199 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:14.972845+0000", 2023-11-01T04:16:15.200 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.141766693, 2023-11-01T04:16:15.200 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10077430499999999, 2023-11-01T04:16:15.200 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.200 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.200 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.201 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.201 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.201 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972845+0000", 2023-11-01T04:16:15.201 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.202 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.202 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.202 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972845+0000", 2023-11-01T04:16:15.202 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.202 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.203 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.203 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972846+0000", 2023-11-01T04:16:15.203 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7299999999999995e-07 2023-11-01T04:16:15.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.203 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.204 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.204 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972858+0000", 2023-11-01T04:16:15.204 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2223e-05 2023-11-01T04:16:15.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.204 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.204 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.205 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972882+0000", 2023-11-01T04:16:15.205 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3683e-05 2023-11-01T04:16:15.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.205 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.205 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.206 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:14.972903+0000", 2023-11-01T04:16:15.206 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0919000000000001e-05 2023-11-01T04:16:15.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.206 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.206 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.207 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.073365+0000", 2023-11-01T04:16:15.207 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100462605 2023-11-01T04:16:15.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.207 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.207 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.073619+0000", 2023-11-01T04:16:15.208 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00025420199999999998 2023-11-01T04:16:15.208 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.208 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.208 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.209 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.209 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.6s5 36/31 ECSubWrite(tid=16, reqid=client.4559.0:113, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2023-11-01T04:16:15.209 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.068661+0000", 2023-11-01T04:16:15.209 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.045950372000000003, 2023-11-01T04:16:15.210 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00077536599999999999, 2023-11-01T04:16:15.210 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.210 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.210 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.210 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.210 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.211 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.068661+0000", 2023-11-01T04:16:15.211 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.211 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.211 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.212 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.068661+0000", 2023-11-01T04:16:15.212 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.212 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.212 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.212 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.068662+0000", 2023-11-01T04:16:15.213 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.026e-06 2023-11-01T04:16:15.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.213 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.213 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.068675+0000", 2023-11-01T04:16:15.213 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2906e-05 2023-11-01T04:16:15.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.214 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.214 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.214 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.068688+0000", 2023-11-01T04:16:15.214 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2935000000000001e-05 2023-11-01T04:16:15.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.215 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.215 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.215 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.068696+0000", 2023-11-01T04:16:15.215 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.7829999999999999e-06 2023-11-01T04:16:15.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.215 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.216 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.216 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.068760+0000", 2023-11-01T04:16:15.216 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3887999999999995e-05 2023-11-01T04:16:15.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.216 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.216 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.217 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.068770+0000", 2023-11-01T04:16:15.217 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.747e-06 2023-11-01T04:16:15.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.217 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.217 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.217 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.069406+0000", 2023-11-01T04:16:15.218 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00063603900000000003 2023-11-01T04:16:15.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.218 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.218 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.218 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.069437+0000", 2023-11-01T04:16:15.218 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1041999999999999e-05 2023-11-01T04:16:15.219 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.219 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.219 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.219 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.219 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.219 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:15.221 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-01T04:16:15.222 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:15.222 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-01T04:16:15.223 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:15.217+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:15.224 INFO:tasks.rados.rados.0.smithi190.stdout:27: finishing write tid 3 to smithi19039307-27 2023-11-01T04:16:15.224 INFO:tasks.rados.rados.0.smithi190.stdout:27: oid 27 updating version 0 to 7 2023-11-01T04:16:15.224 INFO:tasks.rados.rados.0.smithi190.stdout:27: oid 27 updating version 7 to 8 2023-11-01T04:16:15.224 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 27 v 8 (ObjNum 26 snap 0 seq_num 26) dirty exists 2023-11-01T04:16:15.224 INFO:tasks.rados.rados.0.smithi190.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2023-11-01T04:16:15.225 INFO:tasks.rados.rados.0.smithi190.stdout:28: finishing write tid 3 to smithi19039307-28 2023-11-01T04:16:15.225 INFO:tasks.rados.rados.0.smithi190.stdout:28: oid 28 updating version 0 to 9 2023-11-01T04:16:15.225 INFO:tasks.rados.rados.0.smithi190.stdout:28: oid 28 updating version 9 to 10 2023-11-01T04:16:15.225 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 28 v 10 (ObjNum 27 snap 0 seq_num 27) dirty exists 2023-11-01T04:16:15.225 INFO:tasks.rados.rados.0.smithi190.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2023-11-01T04:16:15.225 INFO:tasks.rados.rados.0.smithi190.stdout:27: done (15 left) 2023-11-01T04:16:15.226 INFO:tasks.rados.rados.0.smithi190.stdout:28: done (14 left) 2023-11-01T04:16:15.226 INFO:tasks.rados.rados.0.smithi190.stdout:46: write initial oid 46 2023-11-01T04:16:15.226 INFO:tasks.rados.rados.0.smithi190.stdout:46: seq_num 45 ranges {0=770048} 2023-11-01T04:16:15.226 INFO:tasks.rados.rados.0.smithi190.stdout:46: writing smithi19039307-46 from 0 to 770048 tid 1 2023-11-01T04:16:15.226 INFO:tasks.rados.rados.0.smithi190.stdout:47: write initial oid 47 2023-11-01T04:16:15.227 INFO:tasks.rados.rados.0.smithi190.stdout:47: seq_num 46 ranges {0=786432} 2023-11-01T04:16:15.227 INFO:tasks.rados.rados.0.smithi190.stdout:47: writing smithi19039307-47 from 0 to 786432 tid 1 2023-11-01T04:16:15.227 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:15.227 INFO:tasks.rados.rados.0.smithi190.stdout:36: finishing write tid 3 to smithi19039307-36 2023-11-01T04:16:15.227 INFO:tasks.rados.rados.0.smithi190.stdout:36: oid 36 updating version 0 to 5 2023-11-01T04:16:15.227 INFO:tasks.rados.rados.0.smithi190.stdout:36: oid 36 updating version 5 to 6 2023-11-01T04:16:15.227 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2023-11-01T04:16:15.228 INFO:tasks.rados.rados.0.smithi190.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2023-11-01T04:16:15.228 INFO:tasks.rados.rados.0.smithi190.stdout:24: finishing write tid 3 to smithi19039307-24 2023-11-01T04:16:15.228 INFO:tasks.rados.rados.0.smithi190.stdout:24: oid 24 updating version 0 to 10 2023-11-01T04:16:15.228 INFO:tasks.rados.rados.0.smithi190.stdout:24: oid 24 version 10 is already newer than 9 2023-11-01T04:16:15.228 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 24 v 10 (ObjNum 23 snap 0 seq_num 23) dirty exists 2023-11-01T04:16:15.229 INFO:tasks.rados.rados.0.smithi190.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2023-11-01T04:16:15.229 INFO:tasks.rados.rados.0.smithi190.stdout:41: finishing write tid 2 to smithi19039307-41 2023-11-01T04:16:15.229 INFO:tasks.rados.rados.0.smithi190.stdout:42: finishing write tid 2 to smithi19039307-42 2023-11-01T04:16:15.229 INFO:tasks.rados.rados.0.smithi190.stdout:24: done (15 left) 2023-11-01T04:16:15.229 INFO:tasks.rados.rados.0.smithi190.stdout:36: done (14 left) 2023-11-01T04:16:15.229 INFO:tasks.rados.rados.0.smithi190.stdout:48: write initial oid 48 2023-11-01T04:16:15.230 INFO:tasks.rados.rados.0.smithi190.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2023-11-01T04:16:15.230 INFO:tasks.rados.rados.0.smithi190.stdout:48: writing smithi19039307-48 from 0 to 425984 tid 1 2023-11-01T04:16:15.230 INFO:tasks.rados.rados.0.smithi190.stdout:48: writing smithi19039307-48 from 425984 to 802816 tid 2 2023-11-01T04:16:15.230 INFO:tasks.rados.rados.0.smithi190.stdout:49: write initial oid 49 2023-11-01T04:16:15.230 INFO:tasks.rados.rados.0.smithi190.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2023-11-01T04:16:15.230 INFO:tasks.rados.rados.0.smithi190.stdout:49: writing smithi19039307-49 from 0 to 442368 tid 1 2023-11-01T04:16:15.231 INFO:tasks.rados.rados.0.smithi190.stdout:49: writing smithi19039307-49 from 442368 to 819200 tid 2 2023-11-01T04:16:15.231 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:15.272 INFO:tasks.rados.rados.0.smithi190.stdout:29: finishing write tid 3 to smithi19039307-29 2023-11-01T04:16:15.272 INFO:tasks.rados.rados.0.smithi190.stdout:29: oid 29 updating version 0 to 12 2023-11-01T04:16:15.272 INFO:tasks.rados.rados.0.smithi190.stdout:29: oid 29 version 12 is already newer than 11 2023-11-01T04:16:15.272 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2023-11-01T04:16:15.273 INFO:tasks.rados.rados.0.smithi190.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2023-11-01T04:16:15.273 INFO:tasks.rados.rados.0.smithi190.stdout:29: done (15 left) 2023-11-01T04:16:15.273 INFO:tasks.rados.rados.0.smithi190.stdout:50: write initial oid 50 2023-11-01T04:16:15.273 INFO:tasks.rados.rados.0.smithi190.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2023-11-01T04:16:15.278 INFO:tasks.rados.rados.0.smithi190.stdout:50: writing smithi19039307-50 from 0 to 458752 tid 1 2023-11-01T04:16:15.284 INFO:tasks.rados.rados.0.smithi190.stdout:50: writing smithi19039307-50 from 458752 to 835584 tid 2 2023-11-01T04:16:15.285 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:15.286 INFO:tasks.rados.rados.0.smithi190.stdout:40: finishing write tid 3 to smithi19039307-40 2023-11-01T04:16:15.286 INFO:tasks.rados.rados.0.smithi190.stdout:40: oid 40 updating version 0 to 7 2023-11-01T04:16:15.286 INFO:tasks.rados.rados.0.smithi190.stdout:40: oid 40 updating version 7 to 8 2023-11-01T04:16:15.286 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 40 v 8 (ObjNum 39 snap 0 seq_num 39) dirty exists 2023-11-01T04:16:15.286 INFO:tasks.rados.rados.0.smithi190.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2023-11-01T04:16:15.287 INFO:tasks.rados.rados.0.smithi190.stdout:39: finishing write tid 3 to smithi19039307-39 2023-11-01T04:16:15.287 INFO:tasks.rados.rados.0.smithi190.stdout:39: oid 39 updating version 0 to 5 2023-11-01T04:16:15.287 INFO:tasks.rados.rados.0.smithi190.stdout:39: oid 39 updating version 5 to 6 2023-11-01T04:16:15.287 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2023-11-01T04:16:15.287 INFO:tasks.rados.rados.0.smithi190.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2023-11-01T04:16:15.288 INFO:tasks.rados.rados.0.smithi190.stdout:39: done (15 left) 2023-11-01T04:16:15.288 INFO:tasks.rados.rados.0.smithi190.stdout:40: done (14 left) 2023-11-01T04:16:15.288 INFO:tasks.rados.rados.0.smithi190.stdout:51: copy_from oid 8 from oid 18 current snap is 0 2023-11-01T04:16:15.288 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 24 current snap is 0 2023-11-01T04:16:15.288 INFO:tasks.rados.rados.0.smithi190.stdout:52: seq_num 50 ranges {393216=475136,868352=376832} 2023-11-01T04:16:15.299 INFO:tasks.rados.rados.0.smithi190.stdout:52: writing smithi19039307-24 from 393216 to 868352 tid 1 2023-11-01T04:16:15.306 INFO:tasks.rados.rados.0.smithi190.stdout:52: writing smithi19039307-24 from 868352 to 1245184 tid 2 2023-11-01T04:16:15.306 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:15.306 INFO:tasks.rados.rados.0.smithi190.stdout:33: finishing write tid 3 to smithi19039307-33 2023-11-01T04:16:15.306 INFO:tasks.rados.rados.0.smithi190.stdout:33: oid 33 updating version 0 to 3 2023-11-01T04:16:15.307 INFO:tasks.rados.rados.0.smithi190.stdout:33: oid 33 updating version 3 to 4 2023-11-01T04:16:15.307 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 33 v 4 (ObjNum 32 snap 0 seq_num 32) dirty exists 2023-11-01T04:16:15.307 INFO:tasks.rados.rados.0.smithi190.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2023-11-01T04:16:15.307 INFO:tasks.rados.rados.0.smithi190.stdout:33: done (15 left) 2023-11-01T04:16:15.307 INFO:tasks.rados.rados.0.smithi190.stdout:53: rmattr oid 3 current snap is 0 2023-11-01T04:16:15.307 INFO:tasks.rados.rados.0.smithi190.stdout:53: done (15 left) 2023-11-01T04:16:15.308 INFO:tasks.rados.rados.0.smithi190.stdout:54: delete oid 29 current snap is 0 2023-11-01T04:16:15.321 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:15.317+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:15.335 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:15.336 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-01T04:16:15.373 INFO:tasks.rados.rados.0.smithi190.stdout:37: finishing write tid 3 to smithi19039307-37 2023-11-01T04:16:15.373 INFO:tasks.rados.rados.0.smithi190.stdout:37: oid 37 updating version 0 to 4 2023-11-01T04:16:15.374 INFO:tasks.rados.rados.0.smithi190.stdout:37: oid 37 version 4 is already newer than 3 2023-11-01T04:16:15.374 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 37 v 4 (ObjNum 36 snap 0 seq_num 36) dirty exists 2023-11-01T04:16:15.374 INFO:tasks.rados.rados.0.smithi190.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2023-11-01T04:16:15.377 INFO:tasks.rados.rados.0.smithi190.stdout:45: finishing write tid 1 to smithi19039307-45 2023-11-01T04:16:15.377 INFO:tasks.rados.rados.0.smithi190.stdout:45: finishing write tid 2 to smithi19039307-45 2023-11-01T04:16:15.377 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:15.378 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-01T04:16:15.394 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:15.395 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:16:15.395 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.395 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.as1 36/31 ECSubWrite(tid=24, reqid=client.4559.0:140, at_version=36'4, trim_to=0'0, roll_forward_to=34'2))", 2023-11-01T04:16:15.395 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.199076+0000", 2023-11-01T04:16:15.395 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.196944383, 2023-11-01T04:16:15.396 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19697393299999999, 2023-11-01T04:16:15.396 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.396 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:15.396 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.396 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.396 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.397 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199076+0000", 2023-11-01T04:16:15.397 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.397 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.397 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.397 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199076+0000", 2023-11-01T04:16:15.398 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.398 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.398 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.398 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199078+0000", 2023-11-01T04:16:15.399 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.623e-06 2023-11-01T04:16:15.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.399 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.399 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.399 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199121+0000", 2023-11-01T04:16:15.400 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3727999999999999e-05 2023-11-01T04:16:15.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.400 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.400 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199278+0000", 2023-11-01T04:16:15.400 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015602899999999999 2023-11-01T04:16:15.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.401 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.401 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.401 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299400+0000", 2023-11-01T04:16:15.401 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100122192 2023-11-01T04:16:15.401 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.402 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.402 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.402 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.402 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-11-01T04:16:15.402 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 1 2023-11-01T04:16:15.403 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:15.403 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:15.403 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-01T04:16:15.404 INFO:tasks.rados.rados.0.smithi190.stdout:47: finishing write tid 1 to smithi19039307-47 2023-11-01T04:16:15.404 INFO:tasks.rados.rados.0.smithi190.stdout:41: finishing write tid 3 to smithi19039307-41 2023-11-01T04:16:15.405 INFO:tasks.rados.rados.0.smithi190.stdout:41: oid 41 updating version 0 to 10 2023-11-01T04:16:15.405 INFO:tasks.rados.rados.0.smithi190.stdout:41: oid 41 version 10 is already newer than 9 2023-11-01T04:16:15.405 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 41 v 10 (ObjNum 40 snap 0 seq_num 40) dirty exists 2023-11-01T04:16:15.405 INFO:tasks.rados.rados.0.smithi190.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2023-11-01T04:16:15.410 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-01T04:16:15.416 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:15.416 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-01T04:16:15.422 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:15.422+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:15.440 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:15.440 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-01T04:16:15.471 INFO:tasks.rados.rados.0.smithi190.stdout:38: finishing write tid 3 to smithi19039307-38 2023-11-01T04:16:15.471 INFO:tasks.rados.rados.0.smithi190.stdout:38: oid 38 updating version 0 to 2 2023-11-01T04:16:15.471 INFO:tasks.rados.rados.0.smithi190.stdout:38: oid 38 version 2 is already newer than 1 2023-11-01T04:16:15.471 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 38 v 2 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-11-01T04:16:15.471 INFO:tasks.rados.rados.0.smithi190.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2023-11-01T04:16:15.483 INFO:tasks.rados.rados.0.smithi190.stdout:42: finishing write tid 3 to smithi19039307-42 2023-11-01T04:16:15.483 INFO:tasks.rados.rados.0.smithi190.stdout:42: oid 42 updating version 0 to 6 2023-11-01T04:16:15.483 INFO:tasks.rados.rados.0.smithi190.stdout:42: oid 42 version 6 is already newer than 5 2023-11-01T04:16:15.483 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2023-11-01T04:16:15.483 INFO:tasks.rados.rados.0.smithi190.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2023-11-01T04:16:15.484 INFO:tasks.rados.rados.0.smithi190.stdout:45: finishing write tid 3 to smithi19039307-45 2023-11-01T04:16:15.484 INFO:tasks.rados.rados.0.smithi190.stdout:45: oid 45 updating version 0 to 7 2023-11-01T04:16:15.485 INFO:tasks.rados.rados.0.smithi190.stdout:45: oid 45 updating version 7 to 8 2023-11-01T04:16:15.485 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 45 v 8 (ObjNum 44 snap 0 seq_num 44) dirty exists 2023-11-01T04:16:15.485 INFO:tasks.rados.rados.0.smithi190.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2023-11-01T04:16:15.497 INFO:tasks.rados.rados.0.smithi190.stdout:43: finishing write tid 3 to smithi19039307-43 2023-11-01T04:16:15.497 INFO:tasks.rados.rados.0.smithi190.stdout:43: oid 43 updating version 0 to 5 2023-11-01T04:16:15.497 INFO:tasks.rados.rados.0.smithi190.stdout:43: oid 43 updating version 5 to 6 2023-11-01T04:16:15.498 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2023-11-01T04:16:15.498 INFO:tasks.rados.rados.0.smithi190.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2023-11-01T04:16:15.500 INFO:tasks.rados.rados.0.smithi190.stdout:47: finishing write tid 2 to smithi19039307-47 2023-11-01T04:16:15.504 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:15.505 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-01T04:16:15.522 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:15.519+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:15.526 INFO:teuthology.orchestra.run.smithi153.stdout:osd.12: {} 2023-11-01T04:16:15.526 INFO:teuthology.orchestra.run.smithi153.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-01T04:16:15.533 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:15.533 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:16:15.534 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2023-11-01T04:16:15.534 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2023-11-01T04:16:15.534 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:15.547 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-01T04:16:15.579 INFO:tasks.rados.rados.0.smithi190.stdout:46: finishing write tid 1 to smithi19039307-46 2023-11-01T04:16:15.605 INFO:tasks.rados.rados.0.smithi190.stdout:47: finishing write tid 3 to smithi19039307-47 2023-11-01T04:16:15.605 INFO:tasks.rados.rados.0.smithi190.stdout:47: oid 47 updating version 0 to 4 2023-11-01T04:16:15.605 INFO:tasks.rados.rados.0.smithi190.stdout:47: oid 47 version 4 is already newer than 3 2023-11-01T04:16:15.605 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 47 v 4 (ObjNum 46 snap 0 seq_num 46) dirty exists 2023-11-01T04:16:15.606 INFO:tasks.rados.rados.0.smithi190.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2023-11-01T04:16:15.607 INFO:tasks.rados.rados.0.smithi190.stdout:46: finishing write tid 2 to smithi19039307-46 2023-11-01T04:16:15.618 INFO:tasks.rados.rados.0.smithi190.stdout:50: finishing write tid 1 to smithi19039307-50 2023-11-01T04:16:15.623 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:15.623+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:15.629 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:15.630 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-01T04:16:15.655 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:15.655 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-01T04:16:15.664 INFO:tasks.rados.rados.0.smithi190.stdout:48: finishing write tid 1 to smithi19039307-48 2023-11-01T04:16:15.666 INFO:tasks.rados.rados.0.smithi190.stdout:35: finishing write tid 3 to smithi19039307-35 2023-11-01T04:16:15.666 INFO:tasks.rados.rados.0.smithi190.stdout:35: oid 35 updating version 0 to 12 2023-11-01T04:16:15.666 INFO:tasks.rados.rados.0.smithi190.stdout:35: oid 35 version 12 is already newer than 11 2023-11-01T04:16:15.666 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 35 v 12 (ObjNum 34 snap 0 seq_num 34) dirty exists 2023-11-01T04:16:15.666 INFO:tasks.rados.rados.0.smithi190.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2023-11-01T04:16:15.682 INFO:tasks.rados.rados.0.smithi190.stdout:49: finishing write tid 1 to smithi19039307-49 2023-11-01T04:16:15.682 INFO:tasks.rados.rados.0.smithi190.stdout:49: finishing write tid 2 to smithi19039307-49 2023-11-01T04:16:15.683 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:15.683 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-01T04:16:15.685 INFO:tasks.rados.rados.0.smithi190.stdout:44: finishing write tid 1 to smithi19039307-44 2023-11-01T04:16:15.686 INFO:tasks.rados.rados.0.smithi190.stdout:44: finishing write tid 2 to smithi19039307-44 2023-11-01T04:16:15.704 INFO:tasks.rados.rados.0.smithi190.stdout:52: finishing write tid 1 to smithi19039307-24 2023-11-01T04:16:15.718 INFO:tasks.rados.rados.0.smithi190.stdout:50: finishing write tid 2 to smithi19039307-50 2023-11-01T04:16:15.719 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:15.720 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2023-11-01T04:16:15.720 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2023-11-01T04:16:15.720 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:16:15.720 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.720 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.es3 36/31 ECSubWrite(tid=12, reqid=client.4559.0:133, at_version=36'7, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:15.721 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.169895+0000", 2023-11-01T04:16:15.721 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.54989191100000001, 2023-11-01T04:16:15.721 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103016208, 2023-11-01T04:16:15.721 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.721 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.722 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.722 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.722 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.722 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.169895+0000", 2023-11-01T04:16:15.722 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.722 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.723 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.723 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.723 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.169895+0000", 2023-11-01T04:16:15.723 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.723 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.724 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.724 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.724 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.169897+0000", 2023-11-01T04:16:15.724 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.466e-06 2023-11-01T04:16:15.724 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.724 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.725 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.725 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.170419+0000", 2023-11-01T04:16:15.725 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00052250599999999995 2023-11-01T04:16:15.725 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.725 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.726 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.726 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.170457+0000", 2023-11-01T04:16:15.726 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.7945999999999999e-05 2023-11-01T04:16:15.726 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.726 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.726 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.727 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.170474+0000", 2023-11-01T04:16:15.727 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7045999999999999e-05 2023-11-01T04:16:15.727 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.727 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.727 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.728 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.270728+0000", 2023-11-01T04:16:15.728 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100254038 2023-11-01T04:16:15.728 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.728 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.728 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.729 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.270744+0000", 2023-11-01T04:16:15.729 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.507e-05 2023-11-01T04:16:15.729 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.729 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.729 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.729 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.272841+0000", 2023-11-01T04:16:15.730 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0020970310000000001 2023-11-01T04:16:15.730 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.730 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.730 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.730 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.272912+0000", 2023-11-01T04:16:15.731 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1105000000000004e-05 2023-11-01T04:16:15.731 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.731 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.731 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.731 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.731 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.732 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.es3 36/31 ECSubWrite(tid=13, reqid=client.4559.0:134, at_version=36'8, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:15.732 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.172424+0000", 2023-11-01T04:16:15.732 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.54736311999999998, 2023-11-01T04:16:15.732 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101503802, 2023-11-01T04:16:15.732 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.732 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.733 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.733 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.733 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.733 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.172424+0000", 2023-11-01T04:16:15.733 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.734 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.734 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.734 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.734 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.172424+0000", 2023-11-01T04:16:15.734 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.734 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.735 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.735 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.735 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.172425+0000", 2023-11-01T04:16:15.735 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5600000000000005e-07 2023-11-01T04:16:15.735 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.736 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.736 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.736 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.172438+0000", 2023-11-01T04:16:15.736 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2801e-05 2023-11-01T04:16:15.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.736 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.737 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.737 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.172451+0000", 2023-11-01T04:16:15.737 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3233e-05 2023-11-01T04:16:15.737 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.737 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.737 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.738 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.172462+0000", 2023-11-01T04:16:15.738 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0862e-05 2023-11-01T04:16:15.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.738 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.738 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.739 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.271770+0000", 2023-11-01T04:16:15.739 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.099308117000000001 2023-11-01T04:16:15.739 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.739 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.739 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.739 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.271783+0000", 2023-11-01T04:16:15.740 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2833000000000001e-05 2023-11-01T04:16:15.740 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.740 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.740 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.740 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.273857+0000", 2023-11-01T04:16:15.741 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0020746649999999998 2023-11-01T04:16:15.741 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.741 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.741 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.741 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.273928+0000", 2023-11-01T04:16:15.742 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0635000000000004e-05 2023-11-01T04:16:15.742 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.742 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.742 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.742 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.743 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.743 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.1s1 36/31 ECSubWrite(tid=17, reqid=client.4559.0:122, at_version=0'0, trim_to=0'0, roll_forward_to=35'10))", 2023-11-01T04:16:15.743 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.179353+0000", 2023-11-01T04:16:15.743 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.54043463000000003, 2023-11-01T04:16:15.743 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001017832, 2023-11-01T04:16:15.743 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.744 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.744 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.744 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.744 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.744 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.179353+0000", 2023-11-01T04:16:15.744 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.745 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.745 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.745 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.745 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.179353+0000", 2023-11-01T04:16:15.745 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.745 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.746 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.746 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.746 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.179355+0000", 2023-11-01T04:16:15.746 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.046e-06 2023-11-01T04:16:15.746 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.747 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.747 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.747 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.179383+0000", 2023-11-01T04:16:15.747 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8653000000000002e-05 2023-11-01T04:16:15.747 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.747 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.748 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.748 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.179424+0000", 2023-11-01T04:16:15.748 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0589000000000001e-05 2023-11-01T04:16:15.748 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.748 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.748 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.749 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.179448+0000", 2023-11-01T04:16:15.749 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4414999999999999e-05 2023-11-01T04:16:15.749 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.749 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.749 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.749 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.179542+0000", 2023-11-01T04:16:15.750 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.3221999999999996e-05 2023-11-01T04:16:15.750 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.750 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.750 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.750 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.179555+0000", 2023-11-01T04:16:15.750 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3439e-05 2023-11-01T04:16:15.751 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.751 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.751 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.751 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.180295+0000", 2023-11-01T04:16:15.752 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00073944699999999996 2023-11-01T04:16:15.752 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.752 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.752 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.752 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.180371+0000", 2023-11-01T04:16:15.752 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6020999999999996e-05 2023-11-01T04:16:15.753 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.753 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.753 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.753 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.753 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.753 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.1s1 36/31 ECSubRead(tid=18, to_read={3:872c5fb6:::smithi19039307-41:head=0,4096,0}, subchunks={3:872c5fb6:::smithi19039307-41:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:15.754 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.180245+0000", 2023-11-01T04:16:15.754 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.539542772, 2023-11-01T04:16:15.754 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00038173599999999998, 2023-11-01T04:16:15.754 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.754 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.755 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.755 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.755 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.755 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.180245+0000", 2023-11-01T04:16:15.755 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.755 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.756 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.756 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.756 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.180245+0000", 2023-11-01T04:16:15.756 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.756 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.757 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.757 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.180246+0000", 2023-11-01T04:16:15.757 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.604e-06 2023-11-01T04:16:15.757 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.757 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.757 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.758 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.180265+0000", 2023-11-01T04:16:15.758 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8773000000000001e-05 2023-11-01T04:16:15.758 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.758 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.758 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.758 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.180304+0000", 2023-11-01T04:16:15.759 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8988e-05 2023-11-01T04:16:15.759 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.759 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.759 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.759 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.180336+0000", 2023-11-01T04:16:15.759 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1909000000000002e-05 2023-11-01T04:16:15.760 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.760 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.760 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.760 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.180395+0000", 2023-11-01T04:16:15.761 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.9426000000000003e-05 2023-11-01T04:16:15.761 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.761 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.761 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.761 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.180626+0000", 2023-11-01T04:16:15.762 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000231036 2023-11-01T04:16:15.762 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.762 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.762 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.762 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.762 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.763 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.as1 36/31 ECSubWrite(tid=23, reqid=client.4559.0:139, at_version=36'3, trim_to=0'0, roll_forward_to=34'2))", 2023-11-01T04:16:15.763 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.197284+0000", 2023-11-01T04:16:15.763 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.52250364500000002, 2023-11-01T04:16:15.763 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002324843, 2023-11-01T04:16:15.763 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.764 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.764 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.764 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.764 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.764 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.197284+0000", 2023-11-01T04:16:15.764 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.765 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.765 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.765 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.765 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.197284+0000", 2023-11-01T04:16:15.766 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.766 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.766 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.766 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.766 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.197285+0000", 2023-11-01T04:16:15.766 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.6500000000000008e-07 2023-11-01T04:16:15.767 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.767 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.767 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.767 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.197488+0000", 2023-11-01T04:16:15.767 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00020377899999999999 2023-11-01T04:16:15.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.768 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.768 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.768 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.197509+0000", 2023-11-01T04:16:15.768 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0111000000000001e-05 2023-11-01T04:16:15.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.769 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.769 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.769 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.197519+0000", 2023-11-01T04:16:15.769 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0567999999999999e-05 2023-11-01T04:16:15.769 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.770 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.770 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.770 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.197860+0000", 2023-11-01T04:16:15.770 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00034057500000000002 2023-11-01T04:16:15.770 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.770 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.771 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.771 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.197873+0000", 2023-11-01T04:16:15.771 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3016e-05 2023-11-01T04:16:15.771 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.771 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.771 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.772 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199549+0000", 2023-11-01T04:16:15.772 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0016760740000000001 2023-11-01T04:16:15.772 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.772 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.772 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.772 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199609+0000", 2023-11-01T04:16:15.773 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.9755e-05 2023-11-01T04:16:15.773 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.773 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.773 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.773 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.774 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.as1 36/31 ECSubWrite(tid=24, reqid=client.4559.0:140, at_version=36'4, trim_to=0'0, roll_forward_to=34'2))", 2023-11-01T04:16:15.774 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.199076+0000", 2023-11-01T04:16:15.774 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.52071125399999996, 2023-11-01T04:16:15.774 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20133796800000001, 2023-11-01T04:16:15.774 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.775 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.775 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.775 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.775 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.775 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199076+0000", 2023-11-01T04:16:15.776 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.776 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.776 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.776 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199076+0000", 2023-11-01T04:16:15.776 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.777 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.777 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.777 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.777 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199078+0000", 2023-11-01T04:16:15.777 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.623e-06 2023-11-01T04:16:15.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.778 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.778 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.778 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199121+0000", 2023-11-01T04:16:15.778 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3727999999999999e-05 2023-11-01T04:16:15.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.779 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.779 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.779 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199278+0000", 2023-11-01T04:16:15.779 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015602899999999999 2023-11-01T04:16:15.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.779 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.780 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.780 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299400+0000", 2023-11-01T04:16:15.780 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100122192 2023-11-01T04:16:15.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.780 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.780 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.781 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399577+0000", 2023-11-01T04:16:15.781 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10017696500000001 2023-11-01T04:16:15.781 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.781 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.781 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.781 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399597+0000", 2023-11-01T04:16:15.782 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0213000000000001e-05 2023-11-01T04:16:15.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.782 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.782 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.782 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.400376+0000", 2023-11-01T04:16:15.783 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00077926799999999995 2023-11-01T04:16:15.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.783 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.783 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.783 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.400414+0000", 2023-11-01T04:16:15.783 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.7950000000000001e-05 2023-11-01T04:16:15.784 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.784 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.784 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.784 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.784 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.784 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.es3 36/31 ECSubWrite(tid=14, reqid=client.4559.0:146, at_version=36'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:15.785 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.275417+0000", 2023-11-01T04:16:15.785 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.444370235, 2023-11-01T04:16:15.785 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0016483800000000001, 2023-11-01T04:16:15.785 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.785 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.785 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.786 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.786 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.786 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.275417+0000", 2023-11-01T04:16:15.786 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.787 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.787 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.787 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.275417+0000", 2023-11-01T04:16:15.787 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.787 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.788 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.788 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.275419+0000", 2023-11-01T04:16:15.788 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9400000000000001e-06 2023-11-01T04:16:15.788 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.788 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.788 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.789 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.275684+0000", 2023-11-01T04:16:15.789 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00026499899999999997 2023-11-01T04:16:15.789 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.789 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.789 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.790 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.275719+0000", 2023-11-01T04:16:15.790 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5228999999999998e-05 2023-11-01T04:16:15.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.790 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.790 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.791 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.275745+0000", 2023-11-01T04:16:15.791 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5361999999999999e-05 2023-11-01T04:16:15.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.791 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.791 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.791 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.275902+0000", 2023-11-01T04:16:15.792 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015772900000000001 2023-11-01T04:16:15.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.792 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.792 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.275923+0000", 2023-11-01T04:16:15.792 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0149e-05 2023-11-01T04:16:15.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.793 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.793 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.793 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.277000+0000", 2023-11-01T04:16:15.793 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0010772970000000001 2023-11-01T04:16:15.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.794 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.794 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.794 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.277066+0000", 2023-11-01T04:16:15.794 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5674999999999997e-05 2023-11-01T04:16:15.794 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.794 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.795 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.795 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.795 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.0s2 36/31 ECSubWrite(tid=4, reqid=client.4559.0:136, at_version=36'3, trim_to=0'0, roll_forward_to=33'2))", 2023-11-01T04:16:15.795 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.285003+0000", 2023-11-01T04:16:15.795 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.43478467900000001, 2023-11-01T04:16:15.796 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0023782959999999998, 2023-11-01T04:16:15.796 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.796 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.796 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.796 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.796 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.797 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.285003+0000", 2023-11-01T04:16:15.797 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.797 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.797 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.797 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.285003+0000", 2023-11-01T04:16:15.798 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.798 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.798 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.798 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.798 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.285004+0000", 2023-11-01T04:16:15.799 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.184e-06 2023-11-01T04:16:15.799 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.799 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.799 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.799 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.285888+0000", 2023-11-01T04:16:15.799 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00088414699999999995 2023-11-01T04:16:15.800 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.800 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.800 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.800 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.285913+0000", 2023-11-01T04:16:15.800 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4669e-05 2023-11-01T04:16:15.800 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.801 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.801 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.801 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.285927+0000", 2023-11-01T04:16:15.801 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3927000000000001e-05 2023-11-01T04:16:15.801 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.801 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.802 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.802 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.286022+0000", 2023-11-01T04:16:15.802 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5332000000000001e-05 2023-11-01T04:16:15.802 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.802 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.803 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.803 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.286036+0000", 2023-11-01T04:16:15.803 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4041e-05 2023-11-01T04:16:15.803 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.803 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.803 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.804 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.287338+0000", 2023-11-01T04:16:15.804 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0013020989999999999 2023-11-01T04:16:15.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.804 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.804 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.805 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.287381+0000", 2023-11-01T04:16:15.805 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2896999999999999e-05 2023-11-01T04:16:15.805 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.805 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.805 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.806 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.806 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.0s2 36/31 ECSubWrite(tid=5, reqid=client.4559.0:137, at_version=36'4, trim_to=0'0, roll_forward_to=33'2))", 2023-11-01T04:16:15.806 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.288268+0000", 2023-11-01T04:16:15.806 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.43151899700000002, 2023-11-01T04:16:15.806 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00074284500000000001, 2023-11-01T04:16:15.807 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.807 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.807 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.807 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.807 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.807 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.288268+0000", 2023-11-01T04:16:15.808 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.808 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.808 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.808 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.288268+0000", 2023-11-01T04:16:15.808 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.809 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.809 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.809 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.288269+0000", 2023-11-01T04:16:15.809 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0299999999999997e-07 2023-11-01T04:16:15.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.810 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.810 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.810 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.288295+0000", 2023-11-01T04:16:15.810 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5327e-05 2023-11-01T04:16:15.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.811 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.811 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.811 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.288315+0000", 2023-11-01T04:16:15.811 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.073e-05 2023-11-01T04:16:15.811 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.812 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.812 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.812 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.288330+0000", 2023-11-01T04:16:15.812 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4381e-05 2023-11-01T04:16:15.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.812 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.813 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.813 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.288382+0000", 2023-11-01T04:16:15.813 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1856e-05 2023-11-01T04:16:15.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.813 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.813 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.814 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.288389+0000", 2023-11-01T04:16:15.814 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0249999999999997e-06 2023-11-01T04:16:15.814 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.814 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.814 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.814 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.288984+0000", 2023-11-01T04:16:15.815 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00059494799999999998 2023-11-01T04:16:15.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.815 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.815 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.815 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.289011+0000", 2023-11-01T04:16:15.815 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7674999999999998e-05 2023-11-01T04:16:15.816 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.816 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.816 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.816 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.816 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.ds1 36/31 ECSubRead(tid=25, to_read={3:bc424c1c:::smithi19039307-35:head=0,4096,0}, subchunks={3:bc424c1c:::smithi19039307-35:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:15.817 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.299456+0000", 2023-11-01T04:16:15.817 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.42033140400000002, 2023-11-01T04:16:15.817 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00042498500000000001, 2023-11-01T04:16:15.817 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.817 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.817 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299456+0000", 2023-11-01T04:16:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299456+0000", 2023-11-01T04:16:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.820 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.820 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299457+0000", 2023-11-01T04:16:15.820 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0729999999999999e-06 2023-11-01T04:16:15.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.820 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.821 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.821 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299474+0000", 2023-11-01T04:16:15.821 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6735000000000002e-05 2023-11-01T04:16:15.821 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.821 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.822 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.822 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299493+0000", 2023-11-01T04:16:15.822 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9137e-05 2023-11-01T04:16:15.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.822 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.822 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.823 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299505+0000", 2023-11-01T04:16:15.823 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1691e-05 2023-11-01T04:16:15.823 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.823 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.823 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.824 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299583+0000", 2023-11-01T04:16:15.824 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.8011999999999998e-05 2023-11-01T04:16:15.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.824 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.824 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.824 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299881+0000", 2023-11-01T04:16:15.825 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00029833700000000003 2023-11-01T04:16:15.825 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.825 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.825 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.826 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.826 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds1 36/31 ECSubWrite(tid=26, reqid=client.4559.0:142, at_version=36'13, trim_to=0'0, roll_forward_to=35'12))", 2023-11-01T04:16:15.826 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.299530+0000", 2023-11-01T04:16:15.826 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.42025699500000002, 2023-11-01T04:16:15.826 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0016537170000000001, 2023-11-01T04:16:15.826 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.827 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.827 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.827 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.827 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.827 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299530+0000", 2023-11-01T04:16:15.827 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.828 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.828 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.828 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299530+0000", 2023-11-01T04:16:15.828 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.829 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.829 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.829 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299531+0000", 2023-11-01T04:16:15.829 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.9100000000000004e-07 2023-11-01T04:16:15.830 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.830 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.830 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.830 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299670+0000", 2023-11-01T04:16:15.830 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013883799999999999 2023-11-01T04:16:15.830 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.831 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.831 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.831 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299687+0000", 2023-11-01T04:16:15.831 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6759e-05 2023-11-01T04:16:15.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.832 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.832 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.832 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299695+0000", 2023-11-01T04:16:15.832 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.6109999999999994e-06 2023-11-01T04:16:15.832 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.832 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.833 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.833 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299938+0000", 2023-11-01T04:16:15.833 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000243064 2023-11-01T04:16:15.833 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.833 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.833 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.834 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299948+0000", 2023-11-01T04:16:15.834 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2550000000000003e-06 2023-11-01T04:16:15.834 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.834 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.834 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.835 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.301156+0000", 2023-11-01T04:16:15.835 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001208993 2023-11-01T04:16:15.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.835 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.835 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.836 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.301184+0000", 2023-11-01T04:16:15.836 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7606e-05 2023-11-01T04:16:15.836 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.836 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.836 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.836 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.837 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.837 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s3 36/31 ECSubWrite(tid=186, reqid=client.4559.0:156, at_version=36'13, trim_to=0'0, roll_forward_to=35'12))", 2023-11-01T04:16:15.837 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.310571+0000", 2023-11-01T04:16:15.837 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.40921632099999999, 2023-11-01T04:16:15.837 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001247572, 2023-11-01T04:16:15.838 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.838 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.838 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.838 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.838 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.838 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.310571+0000", 2023-11-01T04:16:15.839 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.839 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.839 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.839 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.310571+0000", 2023-11-01T04:16:15.839 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.840 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.840 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.840 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.310572+0000", 2023-11-01T04:16:15.840 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3349999999999999e-06 2023-11-01T04:16:15.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.841 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.841 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.841 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.310721+0000", 2023-11-01T04:16:15.841 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014818399999999999 2023-11-01T04:16:15.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.842 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.842 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.842 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.310738+0000", 2023-11-01T04:16:15.842 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7669999999999999e-05 2023-11-01T04:16:15.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.843 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.843 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.843 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.310747+0000", 2023-11-01T04:16:15.843 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5520000000000005e-06 2023-11-01T04:16:15.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.844 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.844 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.844 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.310812+0000", 2023-11-01T04:16:15.844 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5110000000000005e-05 2023-11-01T04:16:15.844 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.844 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.845 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.845 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.310818+0000", 2023-11-01T04:16:15.845 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3010000000000003e-06 2023-11-01T04:16:15.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.845 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.846 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.846 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.311773+0000", 2023-11-01T04:16:15.846 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00095516899999999996 2023-11-01T04:16:15.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.846 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.846 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.847 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.311819+0000", 2023-11-01T04:16:15.847 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5250999999999998e-05 2023-11-01T04:16:15.847 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.847 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.847 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.847 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.848 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.848 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s3 36/31 ECSubWrite(tid=187, reqid=client.4559.0:157, at_version=36'14, trim_to=0'0, roll_forward_to=35'12))", 2023-11-01T04:16:15.848 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.314213+0000", 2023-11-01T04:16:15.848 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.40557463300000002, 2023-11-01T04:16:15.848 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0010991729999999999, 2023-11-01T04:16:15.848 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.849 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.849 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.849 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.849 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.849 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.314213+0000", 2023-11-01T04:16:15.849 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.850 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.850 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.850 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.850 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.314213+0000", 2023-11-01T04:16:15.850 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.850 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.851 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.851 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.851 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.314214+0000", 2023-11-01T04:16:15.851 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.026e-06 2023-11-01T04:16:15.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.851 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.852 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.852 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.314299+0000", 2023-11-01T04:16:15.852 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5550000000000006e-05 2023-11-01T04:16:15.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.852 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.853 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.853 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.314313+0000", 2023-11-01T04:16:15.853 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4085e-05 2023-11-01T04:16:15.853 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.853 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.853 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.854 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.314321+0000", 2023-11-01T04:16:15.854 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3459999999999996e-06 2023-11-01T04:16:15.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.854 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.854 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.854 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.314386+0000", 2023-11-01T04:16:15.855 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5382999999999994e-05 2023-11-01T04:16:15.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.855 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.855 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.855 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.314397+0000", 2023-11-01T04:16:15.855 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1106e-05 2023-11-01T04:16:15.856 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.856 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.856 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.856 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.315283+0000", 2023-11-01T04:16:15.856 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00088609599999999995 2023-11-01T04:16:15.856 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.857 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.857 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.857 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.315312+0000", 2023-11-01T04:16:15.857 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8581e-05 2023-11-01T04:16:15.857 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.857 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.858 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.858 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.858 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.858 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.es3 36/31 ECSubRead(tid=15, to_read={3:70b27670:::smithi19039307-45:head=0,4096,0}, subchunks={3:70b27670:::smithi19039307-45:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:15.858 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.379881+0000", 2023-11-01T04:16:15.859 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.33990628299999998, 2023-11-01T04:16:15.859 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00045621399999999998, 2023-11-01T04:16:15.859 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.859 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.859 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.859 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.860 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.860 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.379881+0000", 2023-11-01T04:16:15.860 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.860 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.860 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.860 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.861 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.379881+0000", 2023-11-01T04:16:15.861 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.861 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.861 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.861 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.862 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.379882+0000", 2023-11-01T04:16:15.862 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.223e-06 2023-11-01T04:16:15.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.862 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.862 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.862 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.379898+0000", 2023-11-01T04:16:15.863 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5851000000000001e-05 2023-11-01T04:16:15.863 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.863 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.863 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.863 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.379932+0000", 2023-11-01T04:16:15.863 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.3587000000000002e-05 2023-11-01T04:16:15.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.864 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.864 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.864 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.379949+0000", 2023-11-01T04:16:15.864 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7269e-05 2023-11-01T04:16:15.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.865 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.865 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.865 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.380093+0000", 2023-11-01T04:16:15.865 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000143764 2023-11-01T04:16:15.865 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.865 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.866 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.866 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.380337+0000", 2023-11-01T04:16:15.866 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00024452000000000002 2023-11-01T04:16:15.866 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.866 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.866 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.867 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.867 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.867 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s3 36/31 ECSubWrite(tid=191, reqid=client.4559.0:158, at_version=36'15, trim_to=0'0, roll_forward_to=35'12))", 2023-11-01T04:16:15.867 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.417025+0000", 2023-11-01T04:16:15.867 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.30276241399999998, 2023-11-01T04:16:15.867 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001175588, 2023-11-01T04:16:15.868 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.868 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.868 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.868 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.868 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.868 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417025+0000", 2023-11-01T04:16:15.869 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.869 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.869 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.869 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.869 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417025+0000", 2023-11-01T04:16:15.870 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.870 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.870 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.870 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.870 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417026+0000", 2023-11-01T04:16:15.871 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5099999999999999e-06 2023-11-01T04:16:15.871 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.871 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.871 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.871 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417044+0000", 2023-11-01T04:16:15.871 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7311000000000001e-05 2023-11-01T04:16:15.872 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.872 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.872 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.872 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417062+0000", 2023-11-01T04:16:15.872 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8097999999999999e-05 2023-11-01T04:16:15.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.873 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.873 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.873 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417075+0000", 2023-11-01T04:16:15.873 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.348e-05 2023-11-01T04:16:15.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.874 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.874 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.874 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417181+0000", 2023-11-01T04:16:15.874 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000105504 2023-11-01T04:16:15.874 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.875 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.875 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.875 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417194+0000", 2023-11-01T04:16:15.875 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2677000000000001e-05 2023-11-01T04:16:15.875 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.876 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.876 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.876 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.418175+0000", 2023-11-01T04:16:15.876 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00098175999999999992 2023-11-01T04:16:15.876 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.876 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.877 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.877 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.418201+0000", 2023-11-01T04:16:15.877 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5248e-05 2023-11-01T04:16:15.877 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.877 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.877 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.878 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.878 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.878 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.es3 36/31 ECSubWrite(tid=16, reqid=client.4559.0:147, at_version=36'10, trim_to=0'0, roll_forward_to=36'8))", 2023-11-01T04:16:15.878 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.480851+0000", 2023-11-01T04:16:15.878 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.2389367, 2023-11-01T04:16:15.879 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100941878, 2023-11-01T04:16:15.879 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.879 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.879 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.879 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.879 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.880 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.480851+0000", 2023-11-01T04:16:15.880 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.880 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.880 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.880 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.880 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.480851+0000", 2023-11-01T04:16:15.881 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.881 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.881 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.881 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.480852+0000", 2023-11-01T04:16:15.881 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4139999999999999e-06 2023-11-01T04:16:15.882 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.882 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.882 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.882 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.480951+0000", 2023-11-01T04:16:15.882 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.9147000000000005e-05 2023-11-01T04:16:15.882 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.883 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.883 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.883 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.480973+0000", 2023-11-01T04:16:15.883 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2093e-05 2023-11-01T04:16:15.883 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.883 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.884 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.884 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.480986+0000", 2023-11-01T04:16:15.884 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2855e-05 2023-11-01T04:16:15.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.884 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.884 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.885 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.481057+0000", 2023-11-01T04:16:15.885 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0505000000000003e-05 2023-11-01T04:16:15.885 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.885 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.886 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.886 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.481064+0000", 2023-11-01T04:16:15.886 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4900000000000003e-06 2023-11-01T04:16:15.886 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.886 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.886 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.887 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.581760+0000", 2023-11-01T04:16:15.887 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100696207 2023-11-01T04:16:15.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.887 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.887 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.887 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.581793+0000", 2023-11-01T04:16:15.888 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2166999999999999e-05 2023-11-01T04:16:15.888 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.888 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.888 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.888 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.889 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.889 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.es3 36/31 ECSubWrite(tid=17, reqid=client.4559.0:148, at_version=36'11, trim_to=0'0, roll_forward_to=36'8))", 2023-11-01T04:16:15.889 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.481015+0000", 2023-11-01T04:16:15.889 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.238772763, 2023-11-01T04:16:15.889 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101378417, 2023-11-01T04:16:15.889 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.890 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.890 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.890 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.890 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.890 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.481015+0000", 2023-11-01T04:16:15.891 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.891 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.891 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.891 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.891 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.481015+0000", 2023-11-01T04:16:15.891 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.892 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.892 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.892 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.892 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.481015+0000", 2023-11-01T04:16:15.892 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4999999999999998e-07 2023-11-01T04:16:15.892 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.893 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.893 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.893 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.481032+0000", 2023-11-01T04:16:15.893 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7152000000000001e-05 2023-11-01T04:16:15.893 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.894 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.894 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.894 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.481042+0000", 2023-11-01T04:16:15.894 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0258000000000001e-05 2023-11-01T04:16:15.894 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.894 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.895 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.895 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.481052+0000", 2023-11-01T04:16:15.895 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5319999999999995e-06 2023-11-01T04:16:15.895 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.895 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.895 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.896 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.581560+0000", 2023-11-01T04:16:15.896 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10050753 2023-11-01T04:16:15.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.896 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.896 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.896 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.581569+0000", 2023-11-01T04:16:15.897 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0750000000000004e-06 2023-11-01T04:16:15.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.897 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.897 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.897 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.582354+0000", 2023-11-01T04:16:15.898 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000784952 2023-11-01T04:16:15.898 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.898 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.898 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.898 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.582393+0000", 2023-11-01T04:16:15.898 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9468000000000002e-05 2023-11-01T04:16:15.899 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.899 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.899 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.899 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.899 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.900 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.as1 36/31 ECSubWrite(tid=27, reqid=client.4559.0:140, at_version=0'0, trim_to=0'0, roll_forward_to=36'4))", 2023-11-01T04:16:15.900 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.502207+0000", 2023-11-01T04:16:15.900 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.21758039200000001, 2023-11-01T04:16:15.900 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00081700700000000002, 2023-11-01T04:16:15.900 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.900 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.901 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.901 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.901 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.901 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502207+0000", 2023-11-01T04:16:15.901 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.901 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.902 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.902 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.902 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502207+0000", 2023-11-01T04:16:15.902 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.902 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.902 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.903 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.903 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502209+0000", 2023-11-01T04:16:15.903 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1859999999999999e-06 2023-11-01T04:16:15.903 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.903 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.903 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.904 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502229+0000", 2023-11-01T04:16:15.904 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9973999999999999e-05 2023-11-01T04:16:15.904 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.904 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.904 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.904 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502253+0000", 2023-11-01T04:16:15.905 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3637000000000001e-05 2023-11-01T04:16:15.905 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.905 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.905 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.905 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502265+0000", 2023-11-01T04:16:15.905 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2510000000000001e-05 2023-11-01T04:16:15.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.906 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.906 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.906 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502348+0000", 2023-11-01T04:16:15.906 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3166999999999998e-05 2023-11-01T04:16:15.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.907 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.907 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.907 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502363+0000", 2023-11-01T04:16:15.907 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4737e-05 2023-11-01T04:16:15.907 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.907 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.908 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.908 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502992+0000", 2023-11-01T04:16:15.908 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00062885800000000004 2023-11-01T04:16:15.908 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.908 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.909 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.909 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.503024+0000", 2023-11-01T04:16:15.909 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1937999999999998e-05 2023-11-01T04:16:15.909 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.909 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.909 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.910 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.910 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.910 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.as1 36/31 ECSubRead(tid=28, to_read={3:5f19faeb:::smithi19039307-47:head=0,4096,0}, subchunks={3:5f19faeb:::smithi19039307-47:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:15.910 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.503437+0000", 2023-11-01T04:16:15.910 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.216350508, 2023-11-01T04:16:15.910 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000401383, 2023-11-01T04:16:15.911 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.911 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.911 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.911 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.911 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.911 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.503437+0000", 2023-11-01T04:16:15.912 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.912 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.912 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.912 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.912 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.503437+0000", 2023-11-01T04:16:15.912 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.913 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.913 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.913 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.503438+0000", 2023-11-01T04:16:15.913 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1900000000000002e-07 2023-11-01T04:16:15.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.914 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.914 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.914 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.503448+0000", 2023-11-01T04:16:15.914 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0003999999999999e-05 2023-11-01T04:16:15.914 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.914 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.915 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.915 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.503464+0000", 2023-11-01T04:16:15.915 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6572999999999999e-05 2023-11-01T04:16:15.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.915 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.915 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.916 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.503477+0000", 2023-11-01T04:16:15.916 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2388000000000001e-05 2023-11-01T04:16:15.916 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.916 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.916 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.917 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.503559+0000", 2023-11-01T04:16:15.917 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.2237000000000006e-05 2023-11-01T04:16:15.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.917 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.917 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.917 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.503838+0000", 2023-11-01T04:16:15.918 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00027946200000000002 2023-11-01T04:16:15.918 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.918 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.918 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.918 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.918 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.919 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds1 36/31 ECSubWrite(tid=29, reqid=client.4559.0:142, at_version=0'0, trim_to=0'0, roll_forward_to=36'13))", 2023-11-01T04:16:15.919 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.666550+0000", 2023-11-01T04:16:15.919 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.053237459000000001, 2023-11-01T04:16:15.919 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00094049300000000004, 2023-11-01T04:16:15.919 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:15.920 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:15.920 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:15.920 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.920 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:15.920 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.666550+0000", 2023-11-01T04:16:15.920 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.921 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.921 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.921 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:15.921 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.666550+0000", 2023-11-01T04:16:15.921 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:15.922 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.922 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.922 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:15.922 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.666551+0000", 2023-11-01T04:16:15.922 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.271e-06 2023-11-01T04:16:15.922 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.923 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.923 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:15.923 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.666565+0000", 2023-11-01T04:16:15.923 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4039999999999999e-05 2023-11-01T04:16:15.923 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.924 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.924 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:15.924 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.666586+0000", 2023-11-01T04:16:15.924 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0441999999999998e-05 2023-11-01T04:16:15.924 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.924 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.925 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:15.925 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.666599+0000", 2023-11-01T04:16:15.925 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3580999999999999e-05 2023-11-01T04:16:15.925 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.925 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.926 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:15.926 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.666712+0000", 2023-11-01T04:16:15.926 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000112648 2023-11-01T04:16:15.926 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.926 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.927 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:15.927 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.666720+0000", 2023-11-01T04:16:15.927 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9319999999999996e-06 2023-11-01T04:16:15.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.927 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.927 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:15.928 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.667447+0000", 2023-11-01T04:16:15.928 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00072764200000000002 2023-11-01T04:16:15.928 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:15.928 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:15.928 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:15.929 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.667490+0000", 2023-11-01T04:16:15.929 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2936999999999999e-05 2023-11-01T04:16:15.929 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.929 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.929 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.930 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:15.930 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:15.930 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:15.932 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-01T04:16:15.933 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:15.929+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:15.933 INFO:tasks.rados.rados.0.smithi190.stdout:49: finishing write tid 3 to smithi19039307-49 2023-11-01T04:16:15.933 INFO:tasks.rados.rados.0.smithi190.stdout:44: finishing write tid 3 to smithi19039307-44 2023-11-01T04:16:15.934 INFO:tasks.rados.rados.0.smithi190.stdout:44: oid 44 updating version 0 to 4 2023-11-01T04:16:15.934 INFO:tasks.rados.rados.0.smithi190.stdout:44: oid 44 version 4 is already newer than 3 2023-11-01T04:16:15.934 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 44 v 4 (ObjNum 43 snap 0 seq_num 43) dirty exists 2023-11-01T04:16:15.934 INFO:tasks.rados.rados.0.smithi190.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2023-11-01T04:16:15.934 INFO:tasks.rados.rados.0.smithi190.stdout:52: finishing write tid 2 to smithi19039307-24 2023-11-01T04:16:15.935 INFO:tasks.rados.rados.0.smithi190.stdout:52: finishing write tid 3 to smithi19039307-24 2023-11-01T04:16:15.935 INFO:tasks.rados.rados.0.smithi190.stdout:46: finishing write tid 3 to smithi19039307-46 2023-11-01T04:16:15.935 INFO:tasks.rados.rados.0.smithi190.stdout:46: oid 46 updating version 0 to 4 2023-11-01T04:16:15.935 INFO:tasks.rados.rados.0.smithi190.stdout:46: oid 46 version 4 is already newer than 3 2023-11-01T04:16:15.935 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 46 v 4 (ObjNum 45 snap 0 seq_num 45) dirty exists 2023-11-01T04:16:15.935 INFO:tasks.rados.rados.0.smithi190.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2023-11-01T04:16:15.936 INFO:tasks.rados.rados.0.smithi190.stdout:50: finishing write tid 3 to smithi19039307-50 2023-11-01T04:16:15.936 INFO:tasks.rados.rados.0.smithi190.stdout:48: finishing write tid 2 to smithi19039307-48 2023-11-01T04:16:15.936 INFO:tasks.rados.rados.0.smithi190.stdout:48: finishing write tid 3 to smithi19039307-48 2023-11-01T04:16:15.936 INFO:tasks.rados.rados.0.smithi190.stdout:49: finishing write tid 4 to smithi19039307-49 2023-11-01T04:16:15.936 INFO:tasks.rados.rados.0.smithi190.stdout:49: oid 49 updating version 0 to 9 2023-11-01T04:16:15.937 INFO:tasks.rados.rados.0.smithi190.stdout:49: oid 49 updating version 9 to 11 2023-11-01T04:16:15.937 INFO:tasks.rados.rados.0.smithi190.stdout:49: oid 49 version 11 is already newer than 10 2023-11-01T04:16:15.937 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 49 v 11 (ObjNum 48 snap 0 seq_num 48) dirty exists 2023-11-01T04:16:15.937 INFO:tasks.rados.rados.0.smithi190.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2023-11-01T04:16:15.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-11-01T04:16:16.023 INFO:tasks.rados.rados.0.smithi190.stdout:50: finishing write tid 4 to smithi19039307-50 2023-11-01T04:16:16.023 INFO:tasks.rados.rados.0.smithi190.stdout:50: oid 50 updating version 0 to 6 2023-11-01T04:16:16.023 INFO:tasks.rados.rados.0.smithi190.stdout:50: oid 50 updating version 6 to 7 2023-11-01T04:16:16.024 INFO:tasks.rados.rados.0.smithi190.stdout:50: oid 50 version 7 is already newer than 5 2023-11-01T04:16:16.024 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 50 v 7 (ObjNum 49 snap 0 seq_num 49) dirty exists 2023-11-01T04:16:16.024 INFO:tasks.rados.rados.0.smithi190.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2023-11-01T04:16:16.033 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:16.030+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:16.071 INFO:tasks.rados.rados.0.smithi190.stdout:48: finishing write tid 4 to smithi19039307-48 2023-11-01T04:16:16.071 INFO:tasks.rados.rados.0.smithi190.stdout:48: oid 48 updating version 0 to 13 2023-11-01T04:16:16.071 INFO:tasks.rados.rados.0.smithi190.stdout:48: oid 48 updating version 13 to 14 2023-11-01T04:16:16.071 INFO:tasks.rados.rados.0.smithi190.stdout:48: oid 48 updating version 14 to 15 2023-11-01T04:16:16.072 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 48 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-11-01T04:16:16.072 INFO:tasks.rados.rados.0.smithi190.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2023-11-01T04:16:16.110 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:16.111 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:16:16.111 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2023-11-01T04:16:16.111 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:16.111 INFO:tasks.rados.rados.0.smithi190.stdout:52: finishing write tid 4 to smithi19039307-24 2023-11-01T04:16:16.112 INFO:tasks.rados.rados.0.smithi190.stdout:52: oid 24 updating version 0 to 14 2023-11-01T04:16:16.112 INFO:tasks.rados.rados.0.smithi190.stdout:52: oid 24 version 14 is already newer than 13 2023-11-01T04:16:16.112 INFO:tasks.rados.rados.0.smithi190.stdout:52: oid 24 updating version 14 to 15 2023-11-01T04:16:16.112 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 24 v 15 (ObjNum 50 snap 0 seq_num 50) dirty exists 2023-11-01T04:16:16.112 INFO:tasks.rados.rados.0.smithi190.stdout:52: left oid 24 (ObjNum 50 snap 0 seq_num 50) 2023-11-01T04:16:16.116 INFO:tasks.rados.rados.0.smithi190.stdout:51: finishing copy_from racing read to smithi19039307-8 2023-11-01T04:16:16.131 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-01T04:16:16.133 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:16.129+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:16.218 INFO:tasks.rados.rados.0.smithi190.stdout:51: finishing copy_from to smithi19039307-8 2023-11-01T04:16:16.218 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 8 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-01T04:16:16.234 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:16.232+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:16.245 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:16.245 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:16:16.245 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2023-11-01T04:16:16.245 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2023-11-01T04:16:16.246 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:16.259 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-01T04:16:16.316 INFO:tasks.rados.rados.0.smithi190.stdout:35: done (15 left) 2023-11-01T04:16:16.316 INFO:tasks.rados.rados.0.smithi190.stdout:37: done (14 left) 2023-11-01T04:16:16.316 INFO:tasks.rados.rados.0.smithi190.stdout:38: done (13 left) 2023-11-01T04:16:16.316 INFO:tasks.rados.rados.0.smithi190.stdout:41: done (12 left) 2023-11-01T04:16:16.317 INFO:tasks.rados.rados.0.smithi190.stdout:42: done (11 left) 2023-11-01T04:16:16.317 INFO:tasks.rados.rados.0.smithi190.stdout:43: done (10 left) 2023-11-01T04:16:16.317 INFO:tasks.rados.rados.0.smithi190.stdout:44: done (9 left) 2023-11-01T04:16:16.317 INFO:tasks.rados.rados.0.smithi190.stdout:45: done (8 left) 2023-11-01T04:16:16.318 INFO:tasks.rados.rados.0.smithi190.stdout:46: done (7 left) 2023-11-01T04:16:16.318 INFO:tasks.rados.rados.0.smithi190.stdout:47: done (6 left) 2023-11-01T04:16:16.318 INFO:tasks.rados.rados.0.smithi190.stdout:48: done (5 left) 2023-11-01T04:16:16.318 INFO:tasks.rados.rados.0.smithi190.stdout:49: done (4 left) 2023-11-01T04:16:16.318 INFO:tasks.rados.rados.0.smithi190.stdout:50: done (3 left) 2023-11-01T04:16:16.319 INFO:tasks.rados.rados.0.smithi190.stdout:51: done (2 left) 2023-11-01T04:16:16.319 INFO:tasks.rados.rados.0.smithi190.stdout:52: done (1 left) 2023-11-01T04:16:16.319 INFO:tasks.rados.rados.0.smithi190.stdout:54: done (0 left) 2023-11-01T04:16:16.319 INFO:tasks.rados.rados.0.smithi190.stdout:55: read oid 46 snap -1 2023-11-01T04:16:16.320 INFO:tasks.rados.rados.0.smithi190.stdout:55: expect (ObjNum 45 snap 0 seq_num 45) 2023-11-01T04:16:16.320 INFO:tasks.rados.rados.0.smithi190.stdout:56: rollback oid 29 current snap is 0 2023-11-01T04:16:16.320 INFO:tasks.rados.rados.0.smithi190.stdout:56: done (1 left) 2023-11-01T04:16:16.320 INFO:tasks.rados.rados.0.smithi190.stdout:57: read oid 3 snap -1 2023-11-01T04:16:16.320 INFO:tasks.rados.rados.0.smithi190.stdout:57: expect (ObjNum 2 snap 0 seq_num 2) 2023-11-01T04:16:16.320 INFO:tasks.rados.rados.0.smithi190.stdout:58: copy_from oid 47 from oid 7 current snap is 0 2023-11-01T04:16:16.321 INFO:tasks.rados.rados.0.smithi190.stdout:59: rollback oid 24 current snap is 0 2023-11-01T04:16:16.321 INFO:tasks.rados.rados.0.smithi190.stdout:59: done (3 left) 2023-11-01T04:16:16.321 INFO:tasks.rados.rados.0.smithi190.stdout:60: copy_from oid 35 from oid 48 current snap is 0 2023-11-01T04:16:16.322 INFO:tasks.rados.rados.0.smithi190.stdout:61: delete oid 29 current snap is 0 2023-11-01T04:16:16.335 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:16.336+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:16.380 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:16.380 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2023-11-01T04:16:16.380 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2023-11-01T04:16:16.381 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:16:16.381 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.381 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=186, reqid=client.4559.0:156, at_version=36'13, trim_to=0'0, roll_forward_to=35'12))", 2023-11-01T04:16:16.381 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.399459+0000", 2023-11-01T04:16:16.382 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.979035612, 2023-11-01T04:16:16.382 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.102602861, 2023-11-01T04:16:16.382 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.382 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.382 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.383 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.383 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.383 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399459+0000", 2023-11-01T04:16:16.383 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.384 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.384 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.384 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399459+0000", 2023-11-01T04:16:16.384 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.384 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.385 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.385 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399460+0000", 2023-11-01T04:16:16.385 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3480000000000001e-06 2023-11-01T04:16:16.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.385 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.386 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.386 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399618+0000", 2023-11-01T04:16:16.386 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000158383 2023-11-01T04:16:16.386 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.386 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.387 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.387 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399635+0000", 2023-11-01T04:16:16.387 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6205000000000002e-05 2023-11-01T04:16:16.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.387 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.388 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.388 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399646+0000", 2023-11-01T04:16:16.388 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1274000000000001e-05 2023-11-01T04:16:16.388 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.388 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.389 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.389 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.499871+0000", 2023-11-01T04:16:16.389 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10022505 2023-11-01T04:16:16.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.389 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.390 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.390 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.499887+0000", 2023-11-01T04:16:16.390 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5743000000000002e-05 2023-11-01T04:16:16.390 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.390 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.391 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.391 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502013+0000", 2023-11-01T04:16:16.391 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0021257390000000002 2023-11-01T04:16:16.391 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.391 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.392 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.392 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502062+0000", 2023-11-01T04:16:16.392 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9119000000000002e-05 2023-11-01T04:16:16.392 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.392 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.392 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.393 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.393 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.393 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=187, reqid=client.4559.0:157, at_version=36'14, trim_to=0'0, roll_forward_to=35'12))", 2023-11-01T04:16:16.393 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.399671+0000", 2023-11-01T04:16:16.393 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.97882353600000005, 2023-11-01T04:16:16.394 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10242767899999999, 2023-11-01T04:16:16.394 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.394 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.394 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.394 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.395 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.395 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399671+0000", 2023-11-01T04:16:16.395 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.395 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.395 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.396 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.396 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399671+0000", 2023-11-01T04:16:16.396 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.396 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.396 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.397 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399671+0000", 2023-11-01T04:16:16.397 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2600000000000002e-07 2023-11-01T04:16:16.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.397 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.397 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.398 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399753+0000", 2023-11-01T04:16:16.398 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1821000000000001e-05 2023-11-01T04:16:16.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.398 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.398 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.398 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399761+0000", 2023-11-01T04:16:16.399 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.79e-06 2023-11-01T04:16:16.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.399 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.399 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.399 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399766+0000", 2023-11-01T04:16:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2639999999999999e-06 2023-11-01T04:16:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.400 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.500895+0000", 2023-11-01T04:16:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101128433 2023-11-01T04:16:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.500905+0000", 2023-11-01T04:16:16.401 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0156999999999999e-05 2023-11-01T04:16:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502071+0000", 2023-11-01T04:16:16.402 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001166032 2023-11-01T04:16:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502099+0000", 2023-11-01T04:16:16.403 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7656000000000001e-05 2023-11-01T04:16:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.404 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.3s1 36/31 ECSubWrite(tid=188, reqid=client.4559.0:150, at_version=36'5, trim_to=0'0, roll_forward_to=35'4))", 2023-11-01T04:16:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.416861+0000", 2023-11-01T04:16:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.96163295699999995, 2023-11-01T04:16:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10147307999999999, 2023-11-01T04:16:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.405 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.406 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.416861+0000", 2023-11-01T04:16:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.407 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.416861+0000", 2023-11-01T04:16:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.408 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.416863+0000", 2023-11-01T04:16:16.409 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9999999999999999e-06 2023-11-01T04:16:16.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.409 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.409 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417005+0000", 2023-11-01T04:16:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014113099999999999 2023-11-01T04:16:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.410 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417029+0000", 2023-11-01T04:16:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4031000000000001e-05 2023-11-01T04:16:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.411 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.412 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417045+0000", 2023-11-01T04:16:16.412 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6798000000000001e-05 2023-11-01T04:16:16.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.412 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.412 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.413 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.517210+0000", 2023-11-01T04:16:16.413 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100164471 2023-11-01T04:16:16.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.413 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.413 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.517217+0000", 2023-11-01T04:16:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.737e-06 2023-11-01T04:16:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.415 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.518303+0000", 2023-11-01T04:16:16.415 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001085939 2023-11-01T04:16:16.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.415 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.415 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.415 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.518334+0000", 2023-11-01T04:16:16.416 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1973e-05 2023-11-01T04:16:16.416 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.416 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.416 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.416 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.417 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.3s1 36/31 ECSubWrite(tid=189, reqid=client.4559.0:151, at_version=36'6, trim_to=0'0, roll_forward_to=35'4))", 2023-11-01T04:16:16.417 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.417087+0000", 2023-11-01T04:16:16.417 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.96140752799999996, 2023-11-01T04:16:16.417 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103983992, 2023-11-01T04:16:16.417 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.418 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.418 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.418 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.418 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.418 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417087+0000", 2023-11-01T04:16:16.419 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.419 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.419 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.419 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417087+0000", 2023-11-01T04:16:16.419 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.420 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.420 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.420 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417087+0000", 2023-11-01T04:16:16.420 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2499999999999995e-07 2023-11-01T04:16:16.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.421 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.421 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.421 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417161+0000", 2023-11-01T04:16:16.421 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3508000000000002e-05 2023-11-01T04:16:16.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.422 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.422 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.422 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417171+0000", 2023-11-01T04:16:16.423 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7010000000000008e-06 2023-11-01T04:16:16.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.423 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.423 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.423 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417180+0000", 2023-11-01T04:16:16.424 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2140000000000002e-06 2023-11-01T04:16:16.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.424 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.424 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.424 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.517702+0000", 2023-11-01T04:16:16.425 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100522186 2023-11-01T04:16:16.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.425 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.425 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.425 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.517709+0000", 2023-11-01T04:16:16.425 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0299999999999996e-06 2023-11-01T04:16:16.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.426 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.426 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.426 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.521045+0000", 2023-11-01T04:16:16.426 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0033360640000000001 2023-11-01T04:16:16.427 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.427 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.427 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.427 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.521071+0000", 2023-11-01T04:16:16.427 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5763999999999999e-05 2023-11-01T04:16:16.427 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.428 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.428 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.428 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.428 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.3s1 36/31 ECSubWrite(tid=190, reqid=client.4559.0:152, at_version=36'7, trim_to=0'0, roll_forward_to=35'4))", 2023-11-01T04:16:16.429 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.417198+0000", 2023-11-01T04:16:16.429 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.96129614799999996, 2023-11-01T04:16:16.429 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103890102, 2023-11-01T04:16:16.429 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.429 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.429 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.430 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.430 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.430 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417198+0000", 2023-11-01T04:16:16.430 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.430 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.431 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.431 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.431 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417198+0000", 2023-11-01T04:16:16.431 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.431 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.431 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.432 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.432 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417199+0000", 2023-11-01T04:16:16.432 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1300000000000001e-07 2023-11-01T04:16:16.432 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.432 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.432 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.433 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417214+0000", 2023-11-01T04:16:16.433 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5702999999999999e-05 2023-11-01T04:16:16.433 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.433 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.433 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.433 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417226+0000", 2023-11-01T04:16:16.434 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1602e-05 2023-11-01T04:16:16.434 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.434 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.434 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.434 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417238+0000", 2023-11-01T04:16:16.435 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2133000000000001e-05 2023-11-01T04:16:16.435 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.435 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.435 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.435 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.518027+0000", 2023-11-01T04:16:16.435 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100788603 2023-11-01T04:16:16.436 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.436 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.436 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.436 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.518038+0000", 2023-11-01T04:16:16.436 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0963e-05 2023-11-01T04:16:16.436 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.437 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.437 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.437 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.521074+0000", 2023-11-01T04:16:16.437 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0030366820000000002 2023-11-01T04:16:16.437 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.438 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.438 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.438 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.521088+0000", 2023-11-01T04:16:16.438 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4003e-05 2023-11-01T04:16:16.438 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.439 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.439 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.439 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.439 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.439 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=191, reqid=client.4559.0:158, at_version=36'15, trim_to=0'0, roll_forward_to=35'12))", 2023-11-01T04:16:16.439 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.417254+0000", 2023-11-01T04:16:16.440 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.96124058499999998, 2023-11-01T04:16:16.440 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.085554914999999995, 2023-11-01T04:16:16.440 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.440 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.440 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.440 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.441 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.441 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417254+0000", 2023-11-01T04:16:16.441 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.441 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.441 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.441 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.442 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417254+0000", 2023-11-01T04:16:16.442 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.442 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.442 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.442 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.443 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417254+0000", 2023-11-01T04:16:16.443 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6899999999999998e-07 2023-11-01T04:16:16.443 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.443 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.443 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.443 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417270+0000", 2023-11-01T04:16:16.444 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5614999999999999e-05 2023-11-01T04:16:16.444 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.444 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.444 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.444 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417277+0000", 2023-11-01T04:16:16.444 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5639999999999999e-06 2023-11-01T04:16:16.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.445 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.445 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.445 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417286+0000", 2023-11-01T04:16:16.445 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.2519999999999999e-06 2023-11-01T04:16:16.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.446 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.446 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.446 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.501558+0000", 2023-11-01T04:16:16.446 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.084272044000000004 2023-11-01T04:16:16.446 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.446 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.447 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.447 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.501572+0000", 2023-11-01T04:16:16.447 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4113e-05 2023-11-01T04:16:16.447 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.447 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.447 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.448 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502768+0000", 2023-11-01T04:16:16.448 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0011966749999999999 2023-11-01T04:16:16.448 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.448 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.448 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.449 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502809+0000", 2023-11-01T04:16:16.449 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0283e-05 2023-11-01T04:16:16.449 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.449 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.449 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.449 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.450 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.450 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.6s3 36/31 ECSubWrite(tid=20, reqid=client.4559.0:130, at_version=36'3, trim_to=0'0, roll_forward_to=35'2))", 2023-11-01T04:16:16.450 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.475180+0000", 2023-11-01T04:16:16.450 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.90331454799999999, 2023-11-01T04:16:16.450 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100956516, 2023-11-01T04:16:16.450 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.451 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.451 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.451 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.451 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.451 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.475180+0000", 2023-11-01T04:16:16.452 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.452 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.452 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.452 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.452 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.475180+0000", 2023-11-01T04:16:16.452 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.453 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.453 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.453 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.453 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.475181+0000", 2023-11-01T04:16:16.453 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3939999999999999e-06 2023-11-01T04:16:16.453 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.454 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.454 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.454 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.475677+0000", 2023-11-01T04:16:16.454 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00049528899999999999 2023-11-01T04:16:16.454 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.454 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.455 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.455 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.475700+0000", 2023-11-01T04:16:16.455 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3045e-05 2023-11-01T04:16:16.455 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.455 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.455 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.456 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.475713+0000", 2023-11-01T04:16:16.456 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2948e-05 2023-11-01T04:16:16.456 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.456 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.456 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.456 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.477704+0000", 2023-11-01T04:16:16.457 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001991338 2023-11-01T04:16:16.457 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.457 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.457 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.457 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.477711+0000", 2023-11-01T04:16:16.458 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4749999999999996e-06 2023-11-01T04:16:16.458 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.458 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.458 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.458 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.576102+0000", 2023-11-01T04:16:16.458 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.098391016999999997 2023-11-01T04:16:16.459 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.459 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.459 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.459 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.576136+0000", 2023-11-01T04:16:16.459 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4010000000000001e-05 2023-11-01T04:16:16.459 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.460 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.460 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.460 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.460 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.460 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.6s3 36/31 ECSubWrite(tid=21, reqid=client.4559.0:131, at_version=36'4, trim_to=0'0, roll_forward_to=35'2))", 2023-11-01T04:16:16.460 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.477581+0000", 2023-11-01T04:16:16.461 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.90091384900000004, 2023-11-01T04:16:16.461 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.099492788999999998, 2023-11-01T04:16:16.461 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.461 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.461 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.461 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.462 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.462 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.477581+0000", 2023-11-01T04:16:16.462 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.462 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.462 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.463 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.463 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.477581+0000", 2023-11-01T04:16:16.463 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.463 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.463 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.464 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.464 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.477581+0000", 2023-11-01T04:16:16.464 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4600000000000004e-07 2023-11-01T04:16:16.464 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.464 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.465 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.465 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.477607+0000", 2023-11-01T04:16:16.465 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6145000000000001e-05 2023-11-01T04:16:16.465 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.465 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.465 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.466 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.477624+0000", 2023-11-01T04:16:16.466 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6192e-05 2023-11-01T04:16:16.466 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.466 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.466 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.467 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.477639+0000", 2023-11-01T04:16:16.467 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4997999999999999e-05 2023-11-01T04:16:16.467 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.467 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.467 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.467 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.575894+0000", 2023-11-01T04:16:16.468 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.098255827000000004 2023-11-01T04:16:16.468 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.468 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.468 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.468 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.575903+0000", 2023-11-01T04:16:16.468 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4179999999999997e-06 2023-11-01T04:16:16.469 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.469 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.469 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.469 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.577018+0000", 2023-11-01T04:16:16.469 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0011150859999999999 2023-11-01T04:16:16.469 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.577073+0000", 2023-11-01T04:16:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5377000000000002e-05 2023-11-01T04:16:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.470 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.471 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.471 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.471 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.471 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.as6 36/31 ECSubWrite(tid=27, reqid=client.4559.0:140, at_version=0'0, trim_to=0'0, roll_forward_to=36'4))", 2023-11-01T04:16:16.471 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.502220+0000", 2023-11-01T04:16:16.473 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.87627457499999994, 2023-11-01T04:16:16.473 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0027237609999999999, 2023-11-01T04:16:16.473 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.473 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.474 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.474 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.474 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.474 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502220+0000", 2023-11-01T04:16:16.474 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.474 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.475 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.475 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.475 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502220+0000", 2023-11-01T04:16:16.475 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.475 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.475 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502221+0000", 2023-11-01T04:16:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.04e-06 2023-11-01T04:16:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.476 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.477 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502232+0000", 2023-11-01T04:16:16.477 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0997e-05 2023-11-01T04:16:16.477 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.477 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.477 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.477 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502243+0000", 2023-11-01T04:16:16.478 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1226999999999999e-05 2023-11-01T04:16:16.478 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.478 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.478 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.478 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502253+0000", 2023-11-01T04:16:16.478 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0122000000000001e-05 2023-11-01T04:16:16.479 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.479 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.479 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.479 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502346+0000", 2023-11-01T04:16:16.479 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.3276000000000006e-05 2023-11-01T04:16:16.479 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.480 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.480 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.480 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502357+0000", 2023-11-01T04:16:16.480 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0720000000000001e-05 2023-11-01T04:16:16.480 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.480 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.481 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.481 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.504904+0000", 2023-11-01T04:16:16.481 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002546665 2023-11-01T04:16:16.481 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.481 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.481 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.482 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.504944+0000", 2023-11-01T04:16:16.482 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9714e-05 2023-11-01T04:16:16.482 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.482 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.482 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.482 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.483 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.483 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.6s3 36/31 ECSubWrite(tid=22, reqid=client.4559.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=36'4))", 2023-11-01T04:16:16.483 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.688174+0000", 2023-11-01T04:16:16.483 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.69032033000000004, 2023-11-01T04:16:16.483 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00092446499999999997, 2023-11-01T04:16:16.483 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.484 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.484 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.484 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.484 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.484 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.688174+0000", 2023-11-01T04:16:16.484 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.688174+0000", 2023-11-01T04:16:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.485 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.688176+0000", 2023-11-01T04:16:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.875e-06 2023-11-01T04:16:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.486 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.688200+0000", 2023-11-01T04:16:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4307999999999998e-05 2023-11-01T04:16:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.487 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.688225+0000", 2023-11-01T04:16:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4837e-05 2023-11-01T04:16:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.488 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.688239+0000", 2023-11-01T04:16:16.489 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4398e-05 2023-11-01T04:16:16.489 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.489 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.489 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.489 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.688336+0000", 2023-11-01T04:16:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.6581000000000002e-05 2023-11-01T04:16:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.490 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.688347+0000", 2023-11-01T04:16:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0713999999999999e-05 2023-11-01T04:16:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.689057+0000", 2023-11-01T04:16:16.491 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00071057200000000003 2023-11-01T04:16:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.689099+0000", 2023-11-01T04:16:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1180000000000002e-05 2023-11-01T04:16:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.492 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 36/31 ECSubWrite(tid=29, reqid=client.4559.0:142, at_version=0'0, trim_to=0'0, roll_forward_to=36'13))", 2023-11-01T04:16:16.493 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.709087+0000", 2023-11-01T04:16:16.494 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.66940741999999998, 2023-11-01T04:16:16.494 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0011508270000000001, 2023-11-01T04:16:16.494 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.494 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.494 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.494 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709087+0000", 2023-11-01T04:16:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.495 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.496 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709087+0000", 2023-11-01T04:16:16.496 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.496 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.496 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.496 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.496 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709089+0000", 2023-11-01T04:16:16.497 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7630000000000001e-06 2023-11-01T04:16:16.497 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.497 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.497 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.497 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709109+0000", 2023-11-01T04:16:16.497 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9993e-05 2023-11-01T04:16:16.498 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.498 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.498 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.498 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709134+0000", 2023-11-01T04:16:16.498 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4870000000000001e-05 2023-11-01T04:16:16.498 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.499 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.499 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.499 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709144+0000", 2023-11-01T04:16:16.499 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0695e-05 2023-11-01T04:16:16.499 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.499 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.500 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.500 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709231+0000", 2023-11-01T04:16:16.500 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.7156000000000004e-05 2023-11-01T04:16:16.500 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.501 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.501 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.501 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709239+0000", 2023-11-01T04:16:16.501 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9549999999999992e-06 2023-11-01T04:16:16.501 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.502 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.502 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.502 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.710190+0000", 2023-11-01T04:16:16.502 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00095101599999999995 2023-11-01T04:16:16.502 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.502 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.503 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.503 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.710238+0000", 2023-11-01T04:16:16.503 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7379000000000002e-05 2023-11-01T04:16:16.503 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.503 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.504 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.504 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.504 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.504 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 36/31 ECSubWrite(tid=30, reqid=client.4559.0:143, at_version=36'14, trim_to=0'0, roll_forward_to=36'13))", 2023-11-01T04:16:16.504 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.709173+0000", 2023-11-01T04:16:16.504 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.66932145700000001, 2023-11-01T04:16:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001566897, 2023-11-01T04:16:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.505 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.506 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709173+0000", 2023-11-01T04:16:16.506 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.506 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.506 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.507 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.507 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709173+0000", 2023-11-01T04:16:16.507 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.507 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.507 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.508 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.508 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709173+0000", 2023-11-01T04:16:16.508 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1100000000000001e-07 2023-11-01T04:16:16.508 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.508 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.508 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.509 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709251+0000", 2023-11-01T04:16:16.509 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.7931999999999999e-05 2023-11-01T04:16:16.509 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.509 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.509 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.509 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709264+0000", 2023-11-01T04:16:16.510 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2351000000000001e-05 2023-11-01T04:16:16.510 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.510 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.510 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.510 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709274+0000", 2023-11-01T04:16:16.511 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.9329999999999996e-06 2023-11-01T04:16:16.511 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.511 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.511 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.511 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709510+0000", 2023-11-01T04:16:16.511 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000236321 2023-11-01T04:16:16.512 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.512 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.512 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.512 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709521+0000", 2023-11-01T04:16:16.512 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1603999999999999e-05 2023-11-01T04:16:16.512 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.513 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.513 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.513 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.710697+0000", 2023-11-01T04:16:16.513 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001175953 2023-11-01T04:16:16.513 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.513 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.514 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.514 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.710740+0000", 2023-11-01T04:16:16.514 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2391999999999997e-05 2023-11-01T04:16:16.514 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.514 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.514 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.515 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.515 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.515 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 36/31 ECSubWrite(tid=31, reqid=client.4559.0:144, at_version=36'15, trim_to=0'0, roll_forward_to=36'13))", 2023-11-01T04:16:16.515 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.709296+0000", 2023-11-01T04:16:16.515 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.66919796899999995, 2023-11-01T04:16:16.516 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001808299, 2023-11-01T04:16:16.516 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.516 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.516 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.516 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.517 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.517 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709296+0000", 2023-11-01T04:16:16.517 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.517 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.517 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.517 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.518 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709296+0000", 2023-11-01T04:16:16.518 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.518 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.518 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.518 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.518 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709297+0000", 2023-11-01T04:16:16.519 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2199999999999999e-07 2023-11-01T04:16:16.519 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.519 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.519 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.519 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709313+0000", 2023-11-01T04:16:16.519 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6263999999999999e-05 2023-11-01T04:16:16.520 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.520 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.520 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.520 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709324+0000", 2023-11-01T04:16:16.520 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0862999999999999e-05 2023-11-01T04:16:16.521 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.521 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.521 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.521 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709333+0000", 2023-11-01T04:16:16.521 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.3770000000000002e-06 2023-11-01T04:16:16.521 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.522 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.522 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.522 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.710287+0000", 2023-11-01T04:16:16.522 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00095324400000000001 2023-11-01T04:16:16.522 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.523 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.523 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.523 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.710294+0000", 2023-11-01T04:16:16.523 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0990000000000001e-06 2023-11-01T04:16:16.523 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.523 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.524 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.524 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.711076+0000", 2023-11-01T04:16:16.524 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000782542 2023-11-01T04:16:16.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.524 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.525 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.525 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.711105+0000", 2023-11-01T04:16:16.525 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8487999999999999e-05 2023-11-01T04:16:16.525 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.525 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.525 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.526 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.526 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.526 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=192, reqid=client.4559.0:158, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-11-01T04:16:16.526 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.807815+0000", 2023-11-01T04:16:16.526 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.57067934300000001, 2023-11-01T04:16:16.527 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001475149, 2023-11-01T04:16:16.527 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.527 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.527 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.527 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.528 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.528 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.807815+0000", 2023-11-01T04:16:16.528 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.528 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.528 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.528 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.529 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.807815+0000", 2023-11-01T04:16:16.529 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.529 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.529 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.529 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.530 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.807817+0000", 2023-11-01T04:16:16.530 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9199999999999998e-06 2023-11-01T04:16:16.530 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.530 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.531 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.531 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.807860+0000", 2023-11-01T04:16:16.531 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2718e-05 2023-11-01T04:16:16.531 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.531 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.531 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.532 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.807882+0000", 2023-11-01T04:16:16.532 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2472000000000001e-05 2023-11-01T04:16:16.532 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.532 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.532 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.533 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.807896+0000", 2023-11-01T04:16:16.533 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3776e-05 2023-11-01T04:16:16.533 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.533 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.534 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.534 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.808074+0000", 2023-11-01T04:16:16.534 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00017848199999999999 2023-11-01T04:16:16.534 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.534 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.535 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.535 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.808103+0000", 2023-11-01T04:16:16.535 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8143000000000001e-05 2023-11-01T04:16:16.535 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.535 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.536 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.536 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.809236+0000", 2023-11-01T04:16:16.536 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0011330069999999999 2023-11-01T04:16:16.536 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.536 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.536 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.537 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.809290+0000", 2023-11-01T04:16:16.537 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4630999999999999e-05 2023-11-01T04:16:16.537 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.537 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.537 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.537 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.538 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.538 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.3s1 36/31 ECSubWrite(tid=195, reqid=client.4559.0:152, at_version=0'0, trim_to=0'0, roll_forward_to=36'7))", 2023-11-01T04:16:16.538 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.820923+0000", 2023-11-01T04:16:16.538 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.55757125500000004, 2023-11-01T04:16:16.538 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0010241670000000001, 2023-11-01T04:16:16.539 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.539 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.539 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.539 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.539 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.539 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.820923+0000", 2023-11-01T04:16:16.540 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.540 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.540 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.540 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.540 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.820923+0000", 2023-11-01T04:16:16.541 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.541 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.541 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.541 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.541 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.820924+0000", 2023-11-01T04:16:16.541 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5600000000000004e-07 2023-11-01T04:16:16.542 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.542 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.542 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.542 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.820948+0000", 2023-11-01T04:16:16.542 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3917000000000002e-05 2023-11-01T04:16:16.542 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.543 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.543 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.543 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.820969+0000", 2023-11-01T04:16:16.543 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0767e-05 2023-11-01T04:16:16.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.543 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.544 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.544 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.820986+0000", 2023-11-01T04:16:16.544 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6914000000000001e-05 2023-11-01T04:16:16.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.544 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.544 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.545 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.821065+0000", 2023-11-01T04:16:16.545 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9033e-05 2023-11-01T04:16:16.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.545 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.545 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.545 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.821079+0000", 2023-11-01T04:16:16.546 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4681999999999999e-05 2023-11-01T04:16:16.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.546 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.546 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.546 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.821909+0000", 2023-11-01T04:16:16.547 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000829352 2023-11-01T04:16:16.547 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.547 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.547 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.547 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.821947+0000", 2023-11-01T04:16:16.547 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8546e-05 2023-11-01T04:16:16.548 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.548 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.548 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.548 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.548 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.548 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.3s1 36/31 ECSubRead(tid=196, to_read={3:c87735c0:::smithi19039307-50:head=0,4096,0}, subchunks={3:c87735c0:::smithi19039307-50:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:16.549 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.822326+0000", 2023-11-01T04:16:16.549 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.55616849499999998, 2023-11-01T04:16:16.549 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100663329, 2023-11-01T04:16:16.549 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.549 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.549 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.550 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.550 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.550 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.822326+0000", 2023-11-01T04:16:16.550 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.550 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.551 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.551 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.822326+0000", 2023-11-01T04:16:16.551 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.551 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.551 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.552 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.552 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.822327+0000", 2023-11-01T04:16:16.552 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9500000000000001e-07 2023-11-01T04:16:16.552 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.552 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.552 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.552 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.822338+0000", 2023-11-01T04:16:16.553 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1796e-05 2023-11-01T04:16:16.553 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.553 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.553 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.553 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.822359+0000", 2023-11-01T04:16:16.553 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0690999999999999e-05 2023-11-01T04:16:16.554 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.554 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.554 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.554 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.822372+0000", 2023-11-01T04:16:16.554 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2444e-05 2023-11-01T04:16:16.555 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.555 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.555 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.555 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.922591+0000", 2023-11-01T04:16:16.555 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10021912500000001 2023-11-01T04:16:16.555 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.556 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.556 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.556 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.922989+0000", 2023-11-01T04:16:16.556 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00039847800000000002 2023-11-01T04:16:16.556 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.557 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.557 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.557 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.557 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.557 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 36/31 ECSubWrite(tid=32, reqid=client.4559.0:144, at_version=0'0, trim_to=0'0, roll_forward_to=36'15))", 2023-11-01T04:16:16.557 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.871470+0000", 2023-11-01T04:16:16.557 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.50702391899999999, 2023-11-01T04:16:16.558 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0021396509999999998, 2023-11-01T04:16:16.558 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.558 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.558 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.558 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.559 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.559 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.871470+0000", 2023-11-01T04:16:16.559 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.559 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.559 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.560 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.871470+0000", 2023-11-01T04:16:16.560 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.560 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.560 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.560 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.561 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.871472+0000", 2023-11-01T04:16:16.561 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9470000000000002e-06 2023-11-01T04:16:16.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.561 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.561 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.561 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.871502+0000", 2023-11-01T04:16:16.562 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9322000000000001e-05 2023-11-01T04:16:16.562 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.562 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.562 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.562 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.871542+0000", 2023-11-01T04:16:16.562 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0025999999999999e-05 2023-11-01T04:16:16.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.563 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.563 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.563 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.871565+0000", 2023-11-01T04:16:16.563 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3655999999999998e-05 2023-11-01T04:16:16.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.564 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.564 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.564 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.871812+0000", 2023-11-01T04:16:16.564 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00024626599999999998 2023-11-01T04:16:16.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.565 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.565 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.871827+0000", 2023-11-01T04:16:16.565 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4969e-05 2023-11-01T04:16:16.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.566 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.566 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.873534+0000", 2023-11-01T04:16:16.566 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001707454 2023-11-01T04:16:16.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.566 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.567 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.567 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.873610+0000", 2023-11-01T04:16:16.567 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6011000000000001e-05 2023-11-01T04:16:16.567 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.567 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.567 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.568 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.568 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=197, reqid=client.4559.0:160, at_version=36'16, trim_to=0'0, roll_forward_to=36'15))", 2023-11-01T04:16:16.568 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.114064+0000", 2023-11-01T04:16:16.568 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.26443000900000002, 2023-11-01T04:16:16.568 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101281669, 2023-11-01T04:16:16.569 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.569 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.569 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.569 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.569 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.569 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.114064+0000", 2023-11-01T04:16:16.570 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.570 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.570 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.570 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.114064+0000", 2023-11-01T04:16:16.570 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.571 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.571 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.571 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.114066+0000", 2023-11-01T04:16:16.571 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2100000000000001e-06 2023-11-01T04:16:16.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.572 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.572 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.572 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.114100+0000", 2023-11-01T04:16:16.572 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4758999999999997e-05 2023-11-01T04:16:16.572 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.572 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.573 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.573 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.114125+0000", 2023-11-01T04:16:16.573 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4487000000000001e-05 2023-11-01T04:16:16.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.573 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.574 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.574 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.114136+0000", 2023-11-01T04:16:16.574 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0712e-05 2023-11-01T04:16:16.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.574 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.574 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.575 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.214313+0000", 2023-11-01T04:16:16.575 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10017730699999999 2023-11-01T04:16:16.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.575 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.575 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.575 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.214321+0000", 2023-11-01T04:16:16.576 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5050000000000007e-06 2023-11-01T04:16:16.576 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.576 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.576 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.576 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.215289+0000", 2023-11-01T04:16:16.576 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00096737899999999998 2023-11-01T04:16:16.577 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.577 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.577 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.577 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.215346+0000", 2023-11-01T04:16:16.577 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7309999999999998e-05 2023-11-01T04:16:16.577 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.578 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.578 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.578 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.578 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.578 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 36/31 ECSubWrite(tid=34, reqid=client.4559.0:154, at_version=36'16, trim_to=0'0, roll_forward_to=36'15))", 2023-11-01T04:16:16.578 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.116662+0000", 2023-11-01T04:16:16.579 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.26183216500000001, 2023-11-01T04:16:16.579 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0013599160000000001, 2023-11-01T04:16:16.579 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.579 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.579 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.580 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.580 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.580 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.116662+0000", 2023-11-01T04:16:16.580 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.580 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.580 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.581 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.581 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.116662+0000", 2023-11-01T04:16:16.581 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.581 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.581 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.581 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.582 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.116664+0000", 2023-11-01T04:16:16.582 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6559999999999999e-06 2023-11-01T04:16:16.582 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.582 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.582 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.583 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.116712+0000", 2023-11-01T04:16:16.583 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7961999999999999e-05 2023-11-01T04:16:16.583 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.583 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.583 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.583 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.116737+0000", 2023-11-01T04:16:16.584 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4827000000000001e-05 2023-11-01T04:16:16.584 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.584 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.584 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.584 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.116749+0000", 2023-11-01T04:16:16.584 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2238999999999999e-05 2023-11-01T04:16:16.585 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.585 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.585 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.585 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.116829+0000", 2023-11-01T04:16:16.585 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0081999999999997e-05 2023-11-01T04:16:16.585 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.586 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.586 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.586 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.116839+0000", 2023-11-01T04:16:16.586 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.9380000000000004e-06 2023-11-01T04:16:16.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.586 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.587 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.587 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.117978+0000", 2023-11-01T04:16:16.587 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001138883 2023-11-01T04:16:16.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.587 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.588 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.588 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.118022+0000", 2023-11-01T04:16:16.588 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4329000000000002e-05 2023-11-01T04:16:16.588 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.588 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.588 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.589 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.589 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=198, reqid=client.4559.0:160, at_version=0'0, trim_to=0'0, roll_forward_to=36'16))", 2023-11-01T04:16:16.589 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.317212+0000", 2023-11-01T04:16:16.589 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.061282314999999997, 2023-11-01T04:16:16.589 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00099019300000000011, 2023-11-01T04:16:16.590 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.590 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:16.590 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.590 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.590 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.591 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317212+0000", 2023-11-01T04:16:16.591 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.591 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.591 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.591 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317212+0000", 2023-11-01T04:16:16.592 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.592 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.592 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.592 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317213+0000", 2023-11-01T04:16:16.592 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.418e-06 2023-11-01T04:16:16.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.593 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.593 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.593 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317229+0000", 2023-11-01T04:16:16.593 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5427e-05 2023-11-01T04:16:16.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.594 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.594 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.594 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317253+0000", 2023-11-01T04:16:16.594 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3642000000000002e-05 2023-11-01T04:16:16.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.595 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.595 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.595 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317274+0000", 2023-11-01T04:16:16.595 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1175999999999999e-05 2023-11-01T04:16:16.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.595 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.596 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.596 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317366+0000", 2023-11-01T04:16:16.596 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2644000000000003e-05 2023-11-01T04:16:16.596 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.596 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.597 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.597 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317377+0000", 2023-11-01T04:16:16.597 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0933e-05 2023-11-01T04:16:16.597 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.597 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.597 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.598 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.318160+0000", 2023-11-01T04:16:16.598 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00078288800000000003 2023-11-01T04:16:16.598 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.598 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.598 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:16.598 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.318202+0000", 2023-11-01T04:16:16.599 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2064999999999998e-05 2023-11-01T04:16:16.599 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.599 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.599 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.599 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.599 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.600 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:16.601 INFO:tasks.rados.rados.0.smithi190.stdout:58: finishing copy_from racing read to smithi19039307-47 2023-11-01T04:16:16.602 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:16.602+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:16.602 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-01T04:16:16.620 INFO:tasks.rados.rados.0.smithi190.stdout:61: done (4 left) 2023-11-01T04:16:16.620 INFO:tasks.rados.rados.0.smithi190.stdout:62: read oid 22 snap -1 2023-11-01T04:16:16.620 INFO:tasks.rados.rados.0.smithi190.stdout:62: expect (ObjNum 21 snap 0 seq_num 21) 2023-11-01T04:16:16.621 INFO:tasks.rados.rados.0.smithi190.stdout:63: snap_create 2023-11-01T04:16:16.626 INFO:tasks.rados.rados.0.smithi190.stdout:55: expect (ObjNum 45 snap 0 seq_num 45) 2023-11-01T04:16:16.702 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:16.697+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:16.722 INFO:tasks.rados.rados.0.smithi190.stdout:60: finishing copy_from racing read to smithi19039307-35 2023-11-01T04:16:16.722 INFO:tasks.rados.rados.0.smithi190.stdout:57: expect (ObjNum 2 snap 0 seq_num 2) 2023-11-01T04:16:16.734 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:16.734 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:16:16.734 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.734 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/31 ECSubReadReply(tid=38, attrs_read=0))", 2023-11-01T04:16:16.735 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.624262+0000", 2023-11-01T04:16:16.735 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.10981091900000001, 2023-11-01T04:16:16.735 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.109844778, 2023-11-01T04:16:16.735 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.735 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:16.736 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.736 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.736 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.736 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624262+0000", 2023-11-01T04:16:16.736 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.737 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.737 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.737 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624262+0000", 2023-11-01T04:16:16.737 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.737 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.738 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.738 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.738 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624262+0000", 2023-11-01T04:16:16.738 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.739 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.739 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.739 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624262+0000", 2023-11-01T04:16:16.739 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.739 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.739 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.740 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.740 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.723903+0000", 2023-11-01T04:16:16.740 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.099640944999999995 2023-11-01T04:16:16.740 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.740 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.741 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.741 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.723913+0000", 2023-11-01T04:16:16.741 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5599999999999999e-06 2023-11-01T04:16:16.741 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.741 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.742 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.742 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.742 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.742 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/31 ECSubReadReply(tid=39, attrs_read=0))", 2023-11-01T04:16:16.742 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.624361+0000", 2023-11-01T04:16:16.743 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.109712597, 2023-11-01T04:16:16.743 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10982383799999999, 2023-11-01T04:16:16.743 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.743 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:16.743 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.743 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.744 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.744 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624361+0000", 2023-11-01T04:16:16.744 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.744 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.744 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.745 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.745 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624361+0000", 2023-11-01T04:16:16.745 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.745 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.745 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.746 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.746 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624361+0000", 2023-11-01T04:16:16.746 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.746 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.746 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.747 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.747 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624361+0000", 2023-11-01T04:16:16.747 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.747 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.747 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.748 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.748 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.723932+0000", 2023-11-01T04:16:16.748 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.099571006000000004 2023-11-01T04:16:16.748 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.749 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.749 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.749 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.723938+0000", 2023-11-01T04:16:16.749 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3500000000000002e-06 2023-11-01T04:16:16.749 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.749 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.750 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.750 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.750 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.750 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:169 3.ds0 3:bc424c1c:::smithi19039307-35:head [copy-from ver 15 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:16.750 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.317475+0000", 2023-11-01T04:16:16.751 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.41659809399999997, 2023-11-01T04:16:16.751 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.41679159199999999, 2023-11-01T04:16:16.751 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.751 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:16.751 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:16.751 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:16.752 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:16.752 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 169 2023-11-01T04:16:16.752 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.752 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.752 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.753 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.753 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317475+0000", 2023-11-01T04:16:16.753 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.753 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.753 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.753 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.754 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317475+0000", 2023-11-01T04:16:16.754 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.754 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.754 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.754 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.754 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317476+0000", 2023-11-01T04:16:16.755 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.026e-06 2023-11-01T04:16:16.755 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.755 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.755 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.755 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317482+0000", 2023-11-01T04:16:16.756 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6869999999999999e-06 2023-11-01T04:16:16.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.756 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.756 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.756 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317494+0000", 2023-11-01T04:16:16.756 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2529e-05 2023-11-01T04:16:16.757 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.757 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.757 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.757 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417610+0000", 2023-11-01T04:16:16.757 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100115419 2023-11-01T04:16:16.757 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.758 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.758 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.758 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417723+0000", 2023-11-01T04:16:16.758 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011284499999999999 2023-11-01T04:16:16.758 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.759 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.759 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:16.759 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417815+0000", 2023-11-01T04:16:16.759 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2718000000000002e-05 2023-11-01T04:16:16.759 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.759 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.760 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:16.760 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.630806+0000", 2023-11-01T04:16:16.760 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.21299037200000001 2023-11-01T04:16:16.760 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.760 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.761 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:16.761 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.725358+0000", 2023-11-01T04:16:16.761 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.094552420999999998 2023-11-01T04:16:16.761 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.761 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.762 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.762 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.762 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.762 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/31 ECSubReadReply(tid=39, attrs_read=0))", 2023-11-01T04:16:16.762 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.724544+0000", 2023-11-01T04:16:16.762 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.0095295329999999998, 2023-11-01T04:16:16.763 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0099058089999999998, 2023-11-01T04:16:16.763 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.763 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:16.763 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.763 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.764 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.764 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724544+0000", 2023-11-01T04:16:16.764 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.764 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.764 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.764 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.765 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724544+0000", 2023-11-01T04:16:16.765 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.765 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.765 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.765 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.765 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724545+0000", 2023-11-01T04:16:16.766 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.274e-06 2023-11-01T04:16:16.766 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.766 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.766 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.766 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724748+0000", 2023-11-01T04:16:16.767 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000203117 2023-11-01T04:16:16.767 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.767 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.767 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.767 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724769+0000", 2023-11-01T04:16:16.767 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0588999999999999e-05 2023-11-01T04:16:16.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.768 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.768 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.768 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724783+0000", 2023-11-01T04:16:16.768 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4747e-05 2023-11-01T04:16:16.768 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.769 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.769 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.769 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.769 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.769 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/31 ECSubReadReply(tid=40, attrs_read=0))", 2023-11-01T04:16:16.769 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.724781+0000", 2023-11-01T04:16:16.770 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.0092923429999999998, 2023-11-01T04:16:16.770 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0097674430000000007, 2023-11-01T04:16:16.770 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.770 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:16.770 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.771 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.771 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.771 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724781+0000", 2023-11-01T04:16:16.771 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.771 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.771 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.772 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.772 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724781+0000", 2023-11-01T04:16:16.772 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.772 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.772 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.772 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.773 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724781+0000", 2023-11-01T04:16:16.773 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.773 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.773 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.774 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724781+0000", 2023-11-01T04:16:16.774 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.774 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.774 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.774 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.774 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724823+0000", 2023-11-01T04:16:16.775 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2429000000000003e-05 2023-11-01T04:16:16.775 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.775 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.775 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.775 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724844+0000", 2023-11-01T04:16:16.775 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0305999999999998e-05 2023-11-01T04:16:16.776 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.776 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.776 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.776 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.776 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/31 ECSubReadReply(tid=40, attrs_read=0))", 2023-11-01T04:16:16.777 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.724986+0000", 2023-11-01T04:16:16.777 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.0090870459999999997, 2023-11-01T04:16:16.777 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0096294769999999991, 2023-11-01T04:16:16.777 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.777 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "no flag points reached", 2023-11-01T04:16:16.778 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.778 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.778 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.778 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724986+0000", 2023-11-01T04:16:16.778 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.779 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.779 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.779 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724986+0000", 2023-11-01T04:16:16.779 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.780 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.780 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.780 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724987+0000", 2023-11-01T04:16:16.780 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.114e-06 2023-11-01T04:16:16.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.780 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.781 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.781 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.725128+0000", 2023-11-01T04:16:16.781 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000140362 2023-11-01T04:16:16.781 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.781 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.782 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.782 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.725146+0000", 2023-11-01T04:16:16.782 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8768999999999999e-05 2023-11-01T04:16:16.782 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.782 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.783 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.783 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.783 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 36/31 ECSubWriteReply(tid=41, last_complete=36'17, committed=1, applied=1))", 2023-11-01T04:16:16.783 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.734348+0000", 2023-11-01T04:16:16.784 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 4294967295.9997258, 2023-11-01T04:16:16.784 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00033331000000000002, 2023-11-01T04:16:16.784 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.784 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:16.784 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.784 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.785 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.785 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.734348+0000", 2023-11-01T04:16:16.785 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.785 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.786 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.786 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.734348+0000", 2023-11-01T04:16:16.786 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.786 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.786 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.787 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.734349+0000", 2023-11-01T04:16:16.787 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.437e-06 2023-11-01T04:16:16.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.787 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.787 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.787 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.734356+0000", 2023-11-01T04:16:16.788 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2960000000000001e-06 2023-11-01T04:16:16.788 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.788 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.788 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.788 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.734379+0000", 2023-11-01T04:16:16.788 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2402999999999999e-05 2023-11-01T04:16:16.789 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.789 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.789 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.789 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.734391+0000", 2023-11-01T04:16:16.789 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.183e-05 2023-11-01T04:16:16.790 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.790 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.790 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.790 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.791 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:172 3.ds0 3:b53d1bb4:::smithi19039307-22:head [read 0~360448,checksum in=4b,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:16.791 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.622213+0000", 2023-11-01T04:16:16.791 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.11185980199999999, 2023-11-01T04:16:16.791 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.112548765, 2023-11-01T04:16:16.791 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.791 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:16.792 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:16.792 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:16.792 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:16.792 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 172 2023-11-01T04:16:16.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.793 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.793 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.793 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.793 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622213+0000", 2023-11-01T04:16:16.793 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.794 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.794 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.794 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622213+0000", 2023-11-01T04:16:16.794 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.794 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.794 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.795 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.795 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622216+0000", 2023-11-01T04:16:16.795 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.137e-06 2023-11-01T04:16:16.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.795 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.795 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.796 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622227+0000", 2023-11-01T04:16:16.796 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1861e-05 2023-11-01T04:16:16.796 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.796 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.796 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.796 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622247+0000", 2023-11-01T04:16:16.797 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9656999999999998e-05 2023-11-01T04:16:16.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.797 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.797 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.797 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622259+0000", 2023-11-01T04:16:16.798 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1489e-05 2023-11-01T04:16:16.798 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.798 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.798 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.798 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622359+0000", 2023-11-01T04:16:16.798 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.9995999999999996e-05 2023-11-01T04:16:16.799 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.799 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.799 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:16.799 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622434+0000", 2023-11-01T04:16:16.799 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4975999999999995e-05 2023-11-01T04:16:16.800 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.800 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.800 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.800 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.800 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.800 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:173 3.ds0 3:b53d1bb4:::smithi19039307-22:head [read 0~360448,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:16.801 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.622283+0000", 2023-11-01T04:16:16.801 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.11179006599999999, 2023-11-01T04:16:16.801 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.11258188299999999, 2023-11-01T04:16:16.801 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.801 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:16.802 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:16.802 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:16.802 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:16.802 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 173 2023-11-01T04:16:16.802 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.802 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.803 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.803 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.803 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622283+0000", 2023-11-01T04:16:16.803 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.803 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.803 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.804 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.804 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622283+0000", 2023-11-01T04:16:16.804 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.804 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.804 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.805 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622284+0000", 2023-11-01T04:16:16.805 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7199999999999999e-07 2023-11-01T04:16:16.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.805 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.805 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.805 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622287+0000", 2023-11-01T04:16:16.806 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1879999999999998e-06 2023-11-01T04:16:16.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.806 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.806 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.806 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622295+0000", 2023-11-01T04:16:16.807 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.8820000000000001e-06 2023-11-01T04:16:16.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.807 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.807 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.807 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622300+0000", 2023-11-01T04:16:16.807 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9049999999999996e-06 2023-11-01T04:16:16.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.808 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.808 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.808 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622734+0000", 2023-11-01T04:16:16.808 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0004347 2023-11-01T04:16:16.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.809 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.809 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:16.809 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622866+0000", 2023-11-01T04:16:16.809 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013168300000000001 2023-11-01T04:16:16.809 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.810 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.810 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.810 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.810 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:174 3.ds0 3:b53d1bb4:::smithi19039307-22:head [read 0~360448,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:16.811 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.622306+0000", 2023-11-01T04:16:16.811 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.111767072, 2023-11-01T04:16:16.811 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.11266127200000001, 2023-11-01T04:16:16.811 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:16.811 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:16.811 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:16.812 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:16.812 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:16.812 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 174 2023-11-01T04:16:16.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.812 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:16.812 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.813 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:16.813 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622306+0000", 2023-11-01T04:16:16.813 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.813 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.813 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:16.814 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622306+0000", 2023-11-01T04:16:16.814 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:16.814 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.814 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.814 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:16.815 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622307+0000", 2023-11-01T04:16:16.815 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6899999999999998e-07 2023-11-01T04:16:16.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.815 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.815 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:16.815 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622309+0000", 2023-11-01T04:16:16.816 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3580000000000001e-06 2023-11-01T04:16:16.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.816 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.816 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:16.816 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622315+0000", 2023-11-01T04:16:16.816 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3110000000000002e-06 2023-11-01T04:16:16.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.817 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.817 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:16.817 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622321+0000", 2023-11-01T04:16:16.817 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8000000000000004e-06 2023-11-01T04:16:16.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.818 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.818 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:16.818 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623233+0000", 2023-11-01T04:16:16.818 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00091144800000000003 2023-11-01T04:16:16.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:16.818 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:16.819 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:16.819 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623341+0000", 2023-11-01T04:16:16.819 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000108343 2023-11-01T04:16:16.819 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.819 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:16.820 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.820 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:16.820 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-11-01T04:16:16.820 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 10 2023-11-01T04:16:16.820 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:16.821 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-01T04:16:16.822 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:16.822+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:16.824 INFO:tasks.rados.rados.0.smithi190.stdout:58: finishing copy_from to smithi19039307-47 2023-11-01T04:16:16.824 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 47 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2023-11-01T04:16:16.922 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:16.920+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:16.928 INFO:teuthology.orchestra.run.smithi153.stderr:nodeep-scrub is unset 2023-11-01T04:16:16.937 INFO:tasks.rados.rados.0.smithi190.stdout:55: done (5 left) 2023-11-01T04:16:16.937 INFO:tasks.rados.rados.0.smithi190.stdout:57: done (4 left) 2023-11-01T04:16:16.937 INFO:tasks.rados.rados.0.smithi190.stdout:58: done (3 left) 2023-11-01T04:16:16.937 INFO:tasks.rados.rados.0.smithi190.stdout:63: done (2 left) 2023-11-01T04:16:16.938 INFO:tasks.rados.rados.0.smithi190.stdout:64: copy_from oid 18 from oid 44 current snap is 1 2023-11-01T04:16:16.938 INFO:tasks.rados.rados.0.smithi190.stdout:65: read oid 33 snap -1 2023-11-01T04:16:16.938 INFO:tasks.rados.rados.0.smithi190.stdout:65: expect (ObjNum 32 snap 0 seq_num 32) 2023-11-01T04:16:16.938 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 32 current snap is 1 2023-11-01T04:16:16.938 INFO:tasks.rados.rados.0.smithi190.stdout:66: seq_num 53 ranges {524288=524288,1048576=376832} 2023-11-01T04:16:16.954 INFO:tasks.rados.rados.0.smithi190.stdout:66: writing smithi19039307-32 from 524288 to 1048576 tid 1 2023-11-01T04:16:16.961 INFO:tasks.rados.rados.0.smithi190.stdout:66: writing smithi19039307-32 from 1048576 to 1425408 tid 2 2023-11-01T04:16:16.961 INFO:tasks.rados.rados.0.smithi190.stdout:67: rollback oid 36 current snap is 1 2023-11-01T04:16:16.961 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 36 to 0 2023-11-01T04:16:16.961 INFO:tasks.rados.rados.0.smithi190.stdout:68: setattr oid 34 current snap is 1 2023-11-01T04:16:16.964 INFO:tasks.rados.rados.0.smithi190.stdout:69: copy_from oid 23 from oid 12 current snap is 1 2023-11-01T04:16:16.964 INFO:tasks.rados.rados.0.smithi190.stdout:70: read oid 29 snap 0 2023-11-01T04:16:16.964 INFO:tasks.rados.rados.0.smithi190.stdout:70: expect deleted 2023-11-01T04:16:16.964 INFO:tasks.rados.rados.0.smithi190.stdout:71: copy_from oid 14 from oid 39 current snap is 1 2023-11-01T04:16:16.965 INFO:tasks.rados.rados.0.smithi190.stdout:72: rollback oid 7 current snap is 1 2023-11-01T04:16:16.965 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 7 to 0 2023-11-01T04:16:16.965 INFO:tasks.rados.rados.0.smithi190.stdout:73: read oid 25 snap 0 2023-11-01T04:16:16.965 INFO:tasks.rados.rados.0.smithi190.stdout:73: expect (ObjNum 24 snap 0 seq_num 24) 2023-11-01T04:16:16.965 INFO:tasks.rados.rados.0.smithi190.stdout:74: rmattr oid 16 current snap is 1 2023-11-01T04:16:16.966 INFO:tasks.rados.rados.0.smithi190.stdout:74: done (12 left) 2023-11-01T04:16:16.966 INFO:tasks.rados.rados.0.smithi190.stdout:75: copy_from oid 49 from oid 6 current snap is 1 2023-11-01T04:16:16.966 INFO:tasks.rados.rados.0.smithi190.stdout:76: read oid 10 snap -1 2023-11-01T04:16:16.966 INFO:tasks.rados.rados.0.smithi190.stdout:76: expect (ObjNum 9 snap 0 seq_num 9) 2023-11-01T04:16:16.966 INFO:tasks.rados.rados.0.smithi190.stdout:77: copy_from oid 20 from oid 30 current snap is 1 2023-11-01T04:16:16.966 INFO:tasks.rados.rados.0.smithi190.stdout:78: read oid 37 snap -1 2023-11-01T04:16:16.967 INFO:tasks.rados.rados.0.smithi190.stdout:78: expect (ObjNum 36 snap 0 seq_num 36) 2023-11-01T04:16:16.967 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:16.967 INFO:tasks.rados.rados.0.smithi190.stdout:70: done (15 left) 2023-11-01T04:16:16.968 INFO:tasks.rados.rados.0.smithi190.stdout:79: setattr oid 50 current snap is 1 2023-11-01T04:16:16.968 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 16 2023-11-01T04:16:16.991 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:16.991 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:16:16.991 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2023-11-01T04:16:16.992 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2023-11-01T04:16:16.992 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:17.005 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-01T04:16:17.023 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:17.022+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:17.064 INFO:tasks.rados.rados.0.smithi190.stdout:60: finishing copy_from to smithi19039307-35 2023-11-01T04:16:17.064 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 35 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-11-01T04:16:17.064 INFO:tasks.rados.rados.0.smithi190.stdout:60: done (15 left) 2023-11-01T04:16:17.065 INFO:tasks.rados.rados.0.smithi190.stdout:80: snap_create 2023-11-01T04:16:17.065 INFO:tasks.rados.rados.0.smithi190.stdout:62: expect (ObjNum 21 snap 0 seq_num 21) 2023-11-01T04:16:17.124 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:17.124+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:17.144 INFO:tasks.rados.rados.0.smithi190.stdout:72: finishing rollback tid 0 to smithi19039307-7 2023-11-01T04:16:17.166 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2023-11-01T04:16:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2023-11-01T04:16:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:16:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.as0 36/31 ECSubWriteReply(tid=27, last_complete=36'4, committed=1, applied=1))", 2023-11-01T04:16:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.608783+0000", 2023-11-01T04:16:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 1.5565741559999999, 2023-11-01T04:16:17.168 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100276985, 2023-11-01T04:16:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.169 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.608783+0000", 2023-11-01T04:16:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.170 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.608783+0000", 2023-11-01T04:16:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.171 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.608785+0000", 2023-11-01T04:16:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.409e-06 2023-11-01T04:16:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.172 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.608793+0000", 2023-11-01T04:16:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0979999999999994e-06 2023-11-01T04:16:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.173 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.608808+0000", 2023-11-01T04:16:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4863e-05 2023-11-01T04:16:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.174 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.708921+0000", 2023-11-01T04:16:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10011291 2023-11-01T04:16:17.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.176 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.176 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.176 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709023+0000", 2023-11-01T04:16:17.176 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010238600000000001 2023-11-01T04:16:17.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.177 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.177 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.177 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709060+0000", 2023-11-01T04:16:17.177 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.7319e-05 2023-11-01T04:16:17.178 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.178 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.178 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.178 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.179 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 36/31 ECSubWriteReply(tid=32, last_complete=36'15, committed=1, applied=1))", 2023-11-01T04:16:17.179 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.873120+0000", 2023-11-01T04:16:17.179 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 1.292237605, 2023-11-01T04:16:17.179 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100355259, 2023-11-01T04:16:17.179 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.180 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.180 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.180 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.180 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.181 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.873120+0000", 2023-11-01T04:16:17.181 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.181 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.181 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.182 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.873120+0000", 2023-11-01T04:16:17.182 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.182 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.183 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.183 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.873122+0000", 2023-11-01T04:16:17.183 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.604e-06 2023-11-01T04:16:17.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.183 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.184 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.184 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.873133+0000", 2023-11-01T04:16:17.184 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1292e-05 2023-11-01T04:16:17.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.184 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.184 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.185 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.873165+0000", 2023-11-01T04:16:17.185 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1819999999999997e-05 2023-11-01T04:16:17.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.185 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.186 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.186 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.973305+0000", 2023-11-01T04:16:17.186 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100140162 2023-11-01T04:16:17.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.186 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.187 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.187 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.973445+0000", 2023-11-01T04:16:17.187 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014061499999999999 2023-11-01T04:16:17.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.187 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.188 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.188 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.973475+0000", 2023-11-01T04:16:17.188 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9765999999999999e-05 2023-11-01T04:16:17.188 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.188 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.189 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.189 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/31 ECSubReadReply(tid=33, attrs_read=0))", 2023-11-01T04:16:17.189 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.873641+0000", 2023-11-01T04:16:17.190 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 1.291716927, 2023-11-01T04:16:17.190 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10029165700000001, 2023-11-01T04:16:17.190 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.190 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.191 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.191 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.191 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.873641+0000", 2023-11-01T04:16:17.191 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.192 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.192 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.873641+0000", 2023-11-01T04:16:17.192 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.193 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.193 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.873642+0000", 2023-11-01T04:16:17.193 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2330000000000001e-06 2023-11-01T04:16:17.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.194 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.194 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.194 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.873690+0000", 2023-11-01T04:16:17.194 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7808e-05 2023-11-01T04:16:17.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.195 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.195 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.195 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.873715+0000", 2023-11-01T04:16:17.195 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5072999999999999e-05 2023-11-01T04:16:17.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.196 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.196 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.196 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.973802+0000", 2023-11-01T04:16:17.197 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100087167 2023-11-01T04:16:17.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.197 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.197 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.197 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.973875+0000", 2023-11-01T04:16:17.198 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3287999999999994e-05 2023-11-01T04:16:17.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.198 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.198 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.198 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.973932+0000", 2023-11-01T04:16:17.199 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7088e-05 2023-11-01T04:16:17.199 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.199 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.199 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.200 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.200 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:167 3.as0 3:5f19faeb:::smithi19039307-47:head [copy-from ver 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:17.200 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.317332+0000", 2023-11-01T04:16:17.200 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.84802571699999996, 2023-11-01T04:16:17.201 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.411656464, 2023-11-01T04:16:17.201 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.201 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:17.201 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:17.201 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:17.202 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:17.202 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167 2023-11-01T04:16:17.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.202 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.203 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.203 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.203 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317332+0000", 2023-11-01T04:16:17.203 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.204 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.204 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.204 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317332+0000", 2023-11-01T04:16:17.204 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.205 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.205 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.205 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317334+0000", 2023-11-01T04:16:17.205 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.216e-06 2023-11-01T04:16:17.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.206 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.206 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.206 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317345+0000", 2023-11-01T04:16:17.206 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1381e-05 2023-11-01T04:16:17.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.207 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.207 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.207 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317372+0000", 2023-11-01T04:16:17.207 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6548000000000001e-05 2023-11-01T04:16:17.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.208 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.208 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317385+0000", 2023-11-01T04:16:17.208 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2514000000000001e-05 2023-11-01T04:16:17.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.209 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.209 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.209 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317457+0000", 2023-11-01T04:16:17.209 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2197e-05 2023-11-01T04:16:17.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.210 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.210 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:17.210 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317583+0000", 2023-11-01T04:16:17.210 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000126372 2023-11-01T04:16:17.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.211 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.211 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:17.211 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.524296+0000", 2023-11-01T04:16:17.211 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20671239599999999 2023-11-01T04:16:17.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.212 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.212 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:17.212 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.526421+0000", 2023-11-01T04:16:17.212 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0021255660000000002 2023-11-01T04:16:17.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.213 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.213 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:16:17.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.728850+0000", 2023-11-01T04:16:17.213 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.202429255 2023-11-01T04:16:17.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.214 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.214 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.214 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.728988+0000", 2023-11-01T04:16:17.214 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013801899999999999 2023-11-01T04:16:17.215 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.215 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.215 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.215 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.215 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:168 3.as0 3:5f19faeb:::smithi19039307-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:17.216 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.317417+0000", 2023-11-01T04:16:17.216 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.84794088899999998, 2023-11-01T04:16:17.216 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20846690800000001, 2023-11-01T04:16:17.216 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.216 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:17.217 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:17.217 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:17.217 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:17.217 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 168 2023-11-01T04:16:17.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.217 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.218 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.218 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.218 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317417+0000", 2023-11-01T04:16:17.218 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.219 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.219 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.219 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317417+0000", 2023-11-01T04:16:17.219 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.220 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.220 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.220 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317417+0000", 2023-11-01T04:16:17.220 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3099999999999997e-07 2023-11-01T04:16:17.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.221 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.221 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317420+0000", 2023-11-01T04:16:17.221 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9129999999999999e-06 2023-11-01T04:16:17.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.222 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.222 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.222 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317430+0000", 2023-11-01T04:16:17.222 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.3209999999999994e-06 2023-11-01T04:16:17.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.223 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.223 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.223 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317438+0000", 2023-11-01T04:16:17.223 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.7390000000000005e-06 2023-11-01T04:16:17.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.224 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.224 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.224 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417685+0000", 2023-11-01T04:16:17.224 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10024665100000001 2023-11-01T04:16:17.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.225 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.225 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for blocked object", 2023-11-01T04:16:17.225 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417762+0000", 2023-11-01T04:16:17.225 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.7379000000000006e-05 2023-11-01T04:16:17.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.225 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.226 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.226 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.525430+0000", 2023-11-01T04:16:17.226 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10766753499999999 2023-11-01T04:16:17.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.226 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.227 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:17.227 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.525644+0000", 2023-11-01T04:16:17.227 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00021353999999999999 2023-11-01T04:16:17.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.227 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.227 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.228 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.525884+0000", 2023-11-01T04:16:17.228 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00024009899999999999 2023-11-01T04:16:17.228 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.228 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.228 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.229 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.229 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:170 3.ds0 3:bc424c1c:::smithi19039307-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:17.229 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.417683+0000", 2023-11-01T04:16:17.229 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.74767444900000002, 2023-11-01T04:16:17.229 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30571221100000001, 2023-11-01T04:16:17.230 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.230 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:17.230 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:17.230 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:17.230 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:17.231 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 170 2023-11-01T04:16:17.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.231 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.231 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.231 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.232 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417683+0000", 2023-11-01T04:16:17.232 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.232 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.232 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.232 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417683+0000", 2023-11-01T04:16:17.233 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.233 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.233 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.233 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417685+0000", 2023-11-01T04:16:17.234 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.578e-06 2023-11-01T04:16:17.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.234 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.234 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.234 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417693+0000", 2023-11-01T04:16:17.234 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4810000000000006e-06 2023-11-01T04:16:17.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.235 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.235 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.235 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417706+0000", 2023-11-01T04:16:17.235 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2703e-05 2023-11-01T04:16:17.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.236 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.236 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.236 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417714+0000", 2023-11-01T04:16:17.236 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6890000000000004e-06 2023-11-01T04:16:17.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.237 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.237 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.237 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.418000+0000", 2023-11-01T04:16:17.237 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00028649300000000002 2023-11-01T04:16:17.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.238 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.238 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for blocked object", 2023-11-01T04:16:17.238 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.418160+0000", 2023-11-01T04:16:17.238 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015998600000000001 2023-11-01T04:16:17.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.238 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.239 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.239 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.722848+0000", 2023-11-01T04:16:17.239 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30468759400000001 2023-11-01T04:16:17.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.239 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.240 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:17.240 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.723116+0000", 2023-11-01T04:16:17.240 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00026845200000000002 2023-11-01T04:16:17.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.240 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.240 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.241 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.723395+0000", 2023-11-01T04:16:17.241 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00027923500000000002 2023-11-01T04:16:17.241 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.241 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.241 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.242 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.242 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(osd.12.22:3 3.ds0 3:b31c2ca6:::smithi19039307-48:head [assert-version v15,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:17.242 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.418556+0000", 2023-11-01T04:16:17.242 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.74680138900000004, 2023-11-01T04:16:17.243 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.203260423, 2023-11-01T04:16:17.243 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.243 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:17.243 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.243 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.243 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.244 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.418556+0000", 2023-11-01T04:16:17.244 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.244 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.244 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.244 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.418556+0000", 2023-11-01T04:16:17.245 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.245 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.245 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.245 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.418557+0000", 2023-11-01T04:16:17.246 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.001e-06 2023-11-01T04:16:17.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.246 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.246 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.246 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.418562+0000", 2023-11-01T04:16:17.247 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0590000000000002e-06 2023-11-01T04:16:17.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.247 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.247 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.247 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.418573+0000", 2023-11-01T04:16:17.247 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0390999999999999e-05 2023-11-01T04:16:17.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.248 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.248 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.248 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.418580+0000", 2023-11-01T04:16:17.248 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2969999999999998e-06 2023-11-01T04:16:17.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.249 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.249 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.249 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.418634+0000", 2023-11-01T04:16:17.249 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4007e-05 2023-11-01T04:16:17.249 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.250 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.250 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:17.250 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.418714+0000", 2023-11-01T04:16:17.250 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0034e-05 2023-11-01T04:16:17.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.251 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.251 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.621817+0000", 2023-11-01T04:16:17.251 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.203102634 2023-11-01T04:16:17.251 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.252 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.252 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.252 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.252 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/31 ECSubReadReply(tid=36, attrs_read=0))", 2023-11-01T04:16:17.252 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.520758+0000", 2023-11-01T04:16:17.253 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.64459929299999996, 2023-11-01T04:16:17.253 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101079036, 2023-11-01T04:16:17.253 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.253 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.253 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.254 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.254 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.520758+0000", 2023-11-01T04:16:17.254 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.255 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.255 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.520758+0000", 2023-11-01T04:16:17.255 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.255 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.255 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.256 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.520760+0000", 2023-11-01T04:16:17.256 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.587e-06 2023-11-01T04:16:17.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.256 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.256 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.257 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.521338+0000", 2023-11-01T04:16:17.257 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00057820899999999999 2023-11-01T04:16:17.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.257 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.257 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.257 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.521365+0000", 2023-11-01T04:16:17.258 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7098999999999999e-05 2023-11-01T04:16:17.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.258 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.258 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.258 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.521379+0000", 2023-11-01T04:16:17.259 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.361e-05 2023-11-01T04:16:17.259 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.259 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.259 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.259 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.621552+0000", 2023-11-01T04:16:17.260 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100173263 2023-11-01T04:16:17.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.260 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.260 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.260 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.621837+0000", 2023-11-01T04:16:17.261 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00028526799999999998 2023-11-01T04:16:17.261 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.261 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.261 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.261 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.262 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.262 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.as0 36/31 ECSubWriteReply(tid=37, last_complete=36'5, committed=1, applied=1))", 2023-11-01T04:16:17.262 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.527791+0000", 2023-11-01T04:16:17.262 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.63756682799999997, 2023-11-01T04:16:17.262 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100232051, 2023-11-01T04:16:17.263 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.263 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.263 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.263 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.263 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.264 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.527791+0000", 2023-11-01T04:16:17.264 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.264 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.264 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.264 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.527791+0000", 2023-11-01T04:16:17.265 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.265 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.265 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.265 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.527792+0000", 2023-11-01T04:16:17.265 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.02e-06 2023-11-01T04:16:17.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.266 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.266 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.266 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.527802+0000", 2023-11-01T04:16:17.266 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.8470000000000008e-06 2023-11-01T04:16:17.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.267 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.267 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.267 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.527818+0000", 2023-11-01T04:16:17.267 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6402e-05 2023-11-01T04:16:17.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.268 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.268 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.268 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.627893+0000", 2023-11-01T04:16:17.268 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10007537599999999 2023-11-01T04:16:17.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.268 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.269 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.269 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.628005+0000", 2023-11-01T04:16:17.269 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000111321 2023-11-01T04:16:17.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.269 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.270 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.628023+0000", 2023-11-01T04:16:17.270 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8085e-05 2023-11-01T04:16:17.270 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.270 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.270 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.271 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.271 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:172 3.ds0 3:b53d1bb4:::smithi19039307-22:head [read 0~360448,checksum in=4b,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:17.271 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.622213+0000", 2023-11-01T04:16:17.271 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.54314411500000004, 2023-11-01T04:16:17.271 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20394089400000001, 2023-11-01T04:16:17.272 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.272 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:17.272 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:17.272 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:17.272 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:17.272 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 172 2023-11-01T04:16:17.273 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.273 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.273 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.273 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.273 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622213+0000", 2023-11-01T04:16:17.273 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.274 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.274 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.274 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.274 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622213+0000", 2023-11-01T04:16:17.274 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.275 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.275 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.275 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622216+0000", 2023-11-01T04:16:17.275 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.137e-06 2023-11-01T04:16:17.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.276 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.276 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.276 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622227+0000", 2023-11-01T04:16:17.276 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1861e-05 2023-11-01T04:16:17.276 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.276 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.277 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.277 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622247+0000", 2023-11-01T04:16:17.277 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9656999999999998e-05 2023-11-01T04:16:17.277 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.277 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.277 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.278 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622259+0000", 2023-11-01T04:16:17.278 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1489e-05 2023-11-01T04:16:17.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.278 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.278 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.278 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622359+0000", 2023-11-01T04:16:17.279 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.9995999999999996e-05 2023-11-01T04:16:17.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.279 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:17.279 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622434+0000", 2023-11-01T04:16:17.279 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4975999999999995e-05 2023-11-01T04:16:17.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.280 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.280 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.280 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.826154+0000", 2023-11-01T04:16:17.280 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20372077799999999 2023-11-01T04:16:17.280 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.281 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.281 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.281 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.281 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.ds0 36/31 ECSubRead(tid=40, to_read={3:b53d1bb4:::smithi19039307-22:head=0,90112,0}, subchunks={3:b53d1bb4:::smithi19039307-22:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:17.282 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.623577+0000", 2023-11-01T04:16:17.282 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.54178043899999995, 2023-11-01T04:16:17.282 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101227925, 2023-11-01T04:16:17.282 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.282 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.283 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.283 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.283 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.283 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623577+0000", 2023-11-01T04:16:17.283 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.283 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.284 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.284 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.284 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623577+0000", 2023-11-01T04:16:17.284 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.284 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.285 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.285 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.285 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623577+0000", 2023-11-01T04:16:17.285 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.285 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.285 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.286 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.286 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623577+0000", 2023-11-01T04:16:17.286 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.286 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.286 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.287 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.287 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623726+0000", 2023-11-01T04:16:17.287 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014904999999999999 2023-11-01T04:16:17.287 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.287 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.287 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.288 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.723856+0000", 2023-11-01T04:16:17.288 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100130131 2023-11-01T04:16:17.288 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.288 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.288 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.288 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724523+0000", 2023-11-01T04:16:17.289 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00066694300000000003 2023-11-01T04:16:17.289 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.289 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.289 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.289 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724805+0000", 2023-11-01T04:16:17.290 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00028180099999999998 2023-11-01T04:16:17.290 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.290 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.290 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.290 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.291 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.291 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/31 ECSubReadReply(tid=38, attrs_read=0))", 2023-11-01T04:16:17.291 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.623811+0000", 2023-11-01T04:16:17.291 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.54154639900000001, 2023-11-01T04:16:17.291 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100304453, 2023-11-01T04:16:17.292 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.292 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.292 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.292 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.292 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.293 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623811+0000", 2023-11-01T04:16:17.293 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.293 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.293 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.293 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623811+0000", 2023-11-01T04:16:17.294 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.294 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.294 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.294 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.294 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623812+0000", 2023-11-01T04:16:17.295 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.24e-06 2023-11-01T04:16:17.295 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.295 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.295 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.295 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623907+0000", 2023-11-01T04:16:17.295 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4219e-05 2023-11-01T04:16:17.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.296 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.296 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.296 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623922+0000", 2023-11-01T04:16:17.296 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5696e-05 2023-11-01T04:16:17.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.297 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.297 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.297 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623934+0000", 2023-11-01T04:16:17.297 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1926000000000001e-05 2023-11-01T04:16:17.297 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.298 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.298 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.298 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724032+0000", 2023-11-01T04:16:17.298 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10009749799999999 2023-11-01T04:16:17.298 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.298 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.299 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.299 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724116+0000", 2023-11-01T04:16:17.299 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3874000000000004e-05 2023-11-01T04:16:17.299 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.300 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.300 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.300 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.300 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.300 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/31 ECSubReadReply(tid=39, attrs_read=0))", 2023-11-01T04:16:17.300 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.623961+0000", 2023-11-01T04:16:17.301 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.54139616800000001, 2023-11-01T04:16:17.301 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100465713, 2023-11-01T04:16:17.301 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.301 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.301 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.302 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.302 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.302 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623961+0000", 2023-11-01T04:16:17.302 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.302 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.303 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.303 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.303 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623961+0000", 2023-11-01T04:16:17.303 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.303 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.304 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.304 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.304 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623962+0000", 2023-11-01T04:16:17.304 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1900000000000003e-07 2023-11-01T04:16:17.304 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.305 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.305 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.305 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624052+0000", 2023-11-01T04:16:17.305 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9698999999999996e-05 2023-11-01T04:16:17.305 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.306 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.306 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.306 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624064+0000", 2023-11-01T04:16:17.306 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2169e-05 2023-11-01T04:16:17.306 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.307 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.307 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.307 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624070+0000", 2023-11-01T04:16:17.307 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.852e-06 2023-11-01T04:16:17.307 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.307 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.308 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.308 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724343+0000", 2023-11-01T04:16:17.308 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100273376 2023-11-01T04:16:17.308 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.308 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.309 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.309 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724427+0000", 2023-11-01T04:16:17.309 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4097999999999993e-05 2023-11-01T04:16:17.309 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.309 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.309 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.310 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.310 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/31 ECSubReadReply(tid=40, attrs_read=0))", 2023-11-01T04:16:17.310 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.624084+0000", 2023-11-01T04:16:17.310 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.54127365000000005, 2023-11-01T04:16:17.311 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101138407, 2023-11-01T04:16:17.311 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.311 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.311 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.311 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.312 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.312 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624084+0000", 2023-11-01T04:16:17.312 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.312 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.312 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.313 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.313 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624084+0000", 2023-11-01T04:16:17.313 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.313 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.313 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.313 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.314 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624084+0000", 2023-11-01T04:16:17.314 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.51e-07 2023-11-01T04:16:17.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.314 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.314 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.315 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624192+0000", 2023-11-01T04:16:17.315 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010739300000000001 2023-11-01T04:16:17.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.315 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.315 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.315 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624202+0000", 2023-11-01T04:16:17.316 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0485e-05 2023-11-01T04:16:17.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.316 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.316 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.316 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624211+0000", 2023-11-01T04:16:17.316 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4100000000000008e-06 2023-11-01T04:16:17.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.317 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.317 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.317 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.725109+0000", 2023-11-01T04:16:17.317 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100898615 2023-11-01T04:16:17.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.318 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.318 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.318 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.725222+0000", 2023-11-01T04:16:17.318 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000113053 2023-11-01T04:16:17.318 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.319 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.319 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.319 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.319 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/31 ECSubReadReply(tid=38, attrs_read=0))", 2023-11-01T04:16:17.319 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.624262+0000", 2023-11-01T04:16:17.320 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.54109523199999998, 2023-11-01T04:16:17.320 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.2019252, 2023-11-01T04:16:17.320 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.320 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.320 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.320 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.321 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.321 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624262+0000", 2023-11-01T04:16:17.321 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.321 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.322 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.322 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624262+0000", 2023-11-01T04:16:17.322 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.322 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.322 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.323 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624262+0000", 2023-11-01T04:16:17.323 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.323 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.323 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.323 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624262+0000", 2023-11-01T04:16:17.324 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.324 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.324 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.324 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.324 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.723903+0000", 2023-11-01T04:16:17.325 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.099640944999999995 2023-11-01T04:16:17.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.325 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.325 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.723913+0000", 2023-11-01T04:16:17.325 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5599999999999999e-06 2023-11-01T04:16:17.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.326 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.326 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.326 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.825550+0000", 2023-11-01T04:16:17.326 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101637044 2023-11-01T04:16:17.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.327 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.327 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.327 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.826188+0000", 2023-11-01T04:16:17.327 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00063765100000000004 2023-11-01T04:16:17.327 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.327 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.328 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.328 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.328 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.as0 36/31 ECSubWriteReply(tid=37, last_complete=36'5, committed=1, applied=1))", 2023-11-01T04:16:17.328 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.628558+0000", 2023-11-01T04:16:17.329 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.53679952900000005, 2023-11-01T04:16:17.329 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101148795, 2023-11-01T04:16:17.329 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.329 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.329 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.330 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.330 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.330 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.628558+0000", 2023-11-01T04:16:17.330 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.330 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.331 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.331 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.628558+0000", 2023-11-01T04:16:17.331 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.332 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.332 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.628559+0000", 2023-11-01T04:16:17.332 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.2900000000000002e-07 2023-11-01T04:16:17.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.332 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.332 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.333 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.628565+0000", 2023-11-01T04:16:17.333 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0449999999999997e-06 2023-11-01T04:16:17.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.333 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.333 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.334 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.628583+0000", 2023-11-01T04:16:17.334 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7645000000000001e-05 2023-11-01T04:16:17.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.334 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.335 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.628593+0000", 2023-11-01T04:16:17.335 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0114e-05 2023-11-01T04:16:17.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.335 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.335 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.335 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.728731+0000", 2023-11-01T04:16:17.336 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100138768 2023-11-01T04:16:17.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.336 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.336 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.336 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.729707+0000", 2023-11-01T04:16:17.337 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00097539399999999998 2023-11-01T04:16:17.337 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.337 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.337 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.338 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/31 ECSubReadReply(tid=39, attrs_read=0))", 2023-11-01T04:16:17.338 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.724544+0000", 2023-11-01T04:16:17.338 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.44081384600000001, 2023-11-01T04:16:17.338 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20192078899999999, 2023-11-01T04:16:17.338 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.339 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.339 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.339 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.339 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724544+0000", 2023-11-01T04:16:17.339 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.340 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.340 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724544+0000", 2023-11-01T04:16:17.340 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.341 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.341 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724545+0000", 2023-11-01T04:16:17.341 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.274e-06 2023-11-01T04:16:17.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.342 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.342 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.342 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724748+0000", 2023-11-01T04:16:17.342 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000203117 2023-11-01T04:16:17.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.343 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.343 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724769+0000", 2023-11-01T04:16:17.343 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0588999999999999e-05 2023-11-01T04:16:17.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.344 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.344 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724783+0000", 2023-11-01T04:16:17.344 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4747e-05 2023-11-01T04:16:17.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.344 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.344 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.345 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.926386+0000", 2023-11-01T04:16:17.345 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20160207999999999 2023-11-01T04:16:17.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.345 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.345 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.926465+0000", 2023-11-01T04:16:17.346 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.8981999999999997e-05 2023-11-01T04:16:17.346 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.346 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.346 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.347 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 36/31 ECSubWriteReply(tid=41, last_complete=36'17, committed=1, applied=1))", 2023-11-01T04:16:17.347 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.734348+0000", 2023-11-01T04:16:17.347 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.431010014, 2023-11-01T04:16:17.347 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19222198700000001, 2023-11-01T04:16:17.347 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.348 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.348 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.348 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.348 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.348 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.734348+0000", 2023-11-01T04:16:17.349 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.349 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.349 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.734348+0000", 2023-11-01T04:16:17.349 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.350 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.350 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.734349+0000", 2023-11-01T04:16:17.350 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.437e-06 2023-11-01T04:16:17.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.351 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.351 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.734356+0000", 2023-11-01T04:16:17.351 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2960000000000001e-06 2023-11-01T04:16:17.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.352 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.352 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.352 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.734379+0000", 2023-11-01T04:16:17.352 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2402999999999999e-05 2023-11-01T04:16:17.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.353 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.353 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.353 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.734391+0000", 2023-11-01T04:16:17.353 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.183e-05 2023-11-01T04:16:17.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.354 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.354 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.354 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.926538+0000", 2023-11-01T04:16:17.354 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19214701200000001 2023-11-01T04:16:17.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.354 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.355 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.926570+0000", 2023-11-01T04:16:17.355 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2008999999999998e-05 2023-11-01T04:16:17.355 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.355 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.356 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.356 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.356 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 36/31 ECSubWriteReply(tid=41, last_complete=36'17, committed=1, applied=1))", 2023-11-01T04:16:17.356 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.825290+0000", 2023-11-01T04:16:17.357 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.34006765799999999, 2023-11-01T04:16:17.357 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101365235, 2023-11-01T04:16:17.357 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.357 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.357 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.357 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.358 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.358 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.825290+0000", 2023-11-01T04:16:17.358 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.358 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.358 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.359 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.825290+0000", 2023-11-01T04:16:17.359 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.359 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.359 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.360 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.825291+0000", 2023-11-01T04:16:17.360 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4330000000000001e-06 2023-11-01T04:16:17.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.360 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.360 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.360 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.825301+0000", 2023-11-01T04:16:17.361 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.9799999999999993e-06 2023-11-01T04:16:17.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.361 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.361 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.825324+0000", 2023-11-01T04:16:17.361 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2538999999999999e-05 2023-11-01T04:16:17.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.362 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.362 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.825339+0000", 2023-11-01T04:16:17.362 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5116999999999999e-05 2023-11-01T04:16:17.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.363 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.363 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.926624+0000", 2023-11-01T04:16:17.363 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101285267 2023-11-01T04:16:17.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.364 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.364 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.926655+0000", 2023-11-01T04:16:17.364 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0898999999999997e-05 2023-11-01T04:16:17.364 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.364 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.365 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.365 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.365 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.as0 36/31 ECSubWriteReply(tid=42, last_complete=36'5, committed=1, applied=1))", 2023-11-01T04:16:17.365 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.827289+0000", 2023-11-01T04:16:17.365 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.33806842199999998, 2023-11-01T04:16:17.366 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10032849000000001, 2023-11-01T04:16:17.366 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:17.366 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.366 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:17.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.366 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:17.367 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.827289+0000", 2023-11-01T04:16:17.367 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.367 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:17.367 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.827289+0000", 2023-11-01T04:16:17.368 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:17.368 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.368 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.368 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:17.368 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.827291+0000", 2023-11-01T04:16:17.368 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.64e-06 2023-11-01T04:16:17.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.369 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.369 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:17.369 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.827299+0000", 2023-11-01T04:16:17.369 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5969999999999992e-06 2023-11-01T04:16:17.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.370 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.370 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:17.370 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.827326+0000", 2023-11-01T04:16:17.370 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6852e-05 2023-11-01T04:16:17.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.371 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.371 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.371 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.827340+0000", 2023-11-01T04:16:17.371 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3616e-05 2023-11-01T04:16:17.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.372 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.372 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:17.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.927553+0000", 2023-11-01T04:16:17.372 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10021289 2023-11-01T04:16:17.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:17.372 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:17.373 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:17.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.927618+0000", 2023-11-01T04:16:17.373 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4894999999999994e-05 2023-11-01T04:16:17.373 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.373 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.373 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.374 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:17.374 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:17.374 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:17.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-01T04:16:17.377 INFO:tasks.rados.rados.0.smithi190.stdout:65: expect (ObjNum 32 snap 0 seq_num 32) 2023-11-01T04:16:17.377 INFO:tasks.rados.rados.0.smithi190.stdout:64: finishing copy_from racing read to smithi19039307-18 2023-11-01T04:16:17.377 INFO:tasks.rados.rados.0.smithi190.stdout:69: finishing copy_from racing read to smithi19039307-23 2023-11-01T04:16:17.377 INFO:tasks.rados.rados.0.smithi190.stdout:67: finishing rollback tid 0 to smithi19039307-36 2023-11-01T04:16:17.377 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 34 v 4 (ObjNum 33 snap 0 seq_num 33) dirty exists 2023-11-01T04:16:17.378 INFO:tasks.rados.rados.0.smithi190.stdout:73: expect (ObjNum 24 snap 0 seq_num 24) 2023-11-01T04:16:17.378 INFO:tasks.rados.rados.0.smithi190.stdout:76: expect (ObjNum 9 snap 0 seq_num 9) 2023-11-01T04:16:17.378 INFO:tasks.rados.rados.0.smithi190.stdout:72: finishing rollback tid 1 to smithi19039307-7 2023-11-01T04:16:17.378 INFO:tasks.rados.rados.0.smithi190.stdout:72: finishing rollback tid 2 to smithi19039307-7 2023-11-01T04:16:17.378 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 7 v 10 (ObjNum 6 snap 0 seq_num 6) dirty exists 2023-11-01T04:16:17.379 INFO:tasks.rados.rados.0.smithi190.stdout:67: finishing rollback tid 1 to smithi19039307-36 2023-11-01T04:16:17.379 INFO:tasks.rados.rados.0.smithi190.stdout:67: finishing rollback tid 2 to smithi19039307-36 2023-11-01T04:16:17.379 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 36 v 15 (ObjNum 35 snap 0 seq_num 35) dirty exists 2023-11-01T04:16:17.379 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:17.375+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:17.382 INFO:tasks.rados.rados.0.smithi190.stdout:77: finishing copy_from racing read to smithi19039307-20 2023-11-01T04:16:17.382 INFO:tasks.rados.rados.0.smithi190.stdout:64: finishing copy_from to smithi19039307-18 2023-11-01T04:16:17.383 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 18 v 4 (ObjNum 43 snap 0 seq_num 43) dirty exists 2023-11-01T04:16:17.476 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:17.474+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:17.478 INFO:tasks.rados.rados.0.smithi190.stdout:69: finishing copy_from to smithi19039307-23 2023-11-01T04:16:17.479 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 23 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2023-11-01T04:16:17.481 INFO:tasks.rados.rados.0.smithi190.stdout:71: finishing copy_from racing read to smithi19039307-14 2023-11-01T04:16:17.510 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:17.510 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:17.510 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2023-11-01T04:16:17.511 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:17.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-01T04:16:17.575 INFO:tasks.rados.rados.0.smithi190.stdout:66: finishing write tid 1 to smithi19039307-32 2023-11-01T04:16:17.575 INFO:tasks.rados.rados.0.smithi190.stdout:66: finishing write tid 2 to smithi19039307-32 2023-11-01T04:16:17.576 INFO:tasks.rados.rados.0.smithi190.stdout:66: finishing write tid 3 to smithi19039307-32 2023-11-01T04:16:17.576 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:17.576+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:17.580 INFO:tasks.rados.rados.0.smithi190.stdout:78: expect (ObjNum 36 snap 0 seq_num 36) 2023-11-01T04:16:17.625 INFO:tasks.rados.rados.0.smithi190.stdout:77: finishing copy_from to smithi19039307-20 2023-11-01T04:16:17.626 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 20 v 4 (ObjNum 29 snap 0 seq_num 29) dirty exists 2023-11-01T04:16:17.677 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:17.673+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:17.682 INFO:tasks.rados.rados.0.smithi190.stdout:75: finishing copy_from racing read to smithi19039307-49 2023-11-01T04:16:17.683 INFO:tasks.rados.rados.0.smithi190.stdout:66: finishing write tid 4 to smithi19039307-32 2023-11-01T04:16:17.683 INFO:tasks.rados.rados.0.smithi190.stdout:66: oid 32 updating version 0 to 9 2023-11-01T04:16:17.684 INFO:tasks.rados.rados.0.smithi190.stdout:66: oid 32 updating version 9 to 10 2023-11-01T04:16:17.684 INFO:tasks.rados.rados.0.smithi190.stdout:66: oid 32 version 10 is already newer than 8 2023-11-01T04:16:17.684 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 32 v 10 (ObjNum 53 snap 1 seq_num 53) dirty exists 2023-11-01T04:16:17.684 INFO:tasks.rados.rados.0.smithi190.stdout:66: left oid 32 (ObjNum 53 snap 1 seq_num 53) 2023-11-01T04:16:17.686 INFO:tasks.rados.rados.0.smithi190.stdout:71: finishing copy_from to smithi19039307-14 2023-11-01T04:16:17.686 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 14 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2023-11-01T04:16:17.687 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:17.688 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:17.688 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:16:17.688 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:16:17.688 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:17.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-01T04:16:17.706 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-01T04:16:17.778 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:17.778+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:17.827 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:17.827 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:16:17.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:16:17.828 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:16:17.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.828 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.6s2 36/31 ECSubWrite(tid=16, reqid=client.4559.0:113, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2023-11-01T04:16:17.829 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.068503+0000", 2023-11-01T04:16:17.829 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.7568444369999998, 2023-11-01T04:16:17.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10200353700000001, 2023-11-01T04:16:17.829 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:17.830 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.830 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:17.830 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.830 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:17.830 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.068503+0000", 2023-11-01T04:16:17.831 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.831 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.831 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:17.832 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.068503+0000", 2023-11-01T04:16:17.832 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.832 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.832 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:17.833 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.068504+0000", 2023-11-01T04:16:17.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5999999999999999e-06 2023-11-01T04:16:17.833 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.833 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:17.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.068528+0000", 2023-11-01T04:16:17.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3799e-05 2023-11-01T04:16:17.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:17.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.068562+0000", 2023-11-01T04:16:17.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4319000000000001e-05 2023-11-01T04:16:17.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.168742+0000", 2023-11-01T04:16:17.836 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10017962399999999 2023-11-01T04:16:17.836 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.836 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.836 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:17.836 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.168891+0000", 2023-11-01T04:16:17.837 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014886700000000001 2023-11-01T04:16:17.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.837 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:17.837 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.168921+0000", 2023-11-01T04:16:17.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9882e-05 2023-11-01T04:16:17.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:17.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.170470+0000", 2023-11-01T04:16:17.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015498109999999999 2023-11-01T04:16:17.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.839 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.839 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:17.839 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.170506+0000", 2023-11-01T04:16:17.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5635e-05 2023-11-01T04:16:17.840 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.840 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:17.840 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.841 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.as2 36/31 ECSubWrite(tid=23, reqid=client.4559.0:139, at_version=36'3, trim_to=0'0, roll_forward_to=34'2))", 2023-11-01T04:16:17.841 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.196944+0000", 2023-11-01T04:16:17.841 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.6284032330000002, 2023-11-01T04:16:17.841 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102992326, 2023-11-01T04:16:17.841 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:17.842 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.842 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:17.842 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.842 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:17.842 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.196944+0000", 2023-11-01T04:16:17.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.843 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.843 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.843 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:17.843 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.196944+0000", 2023-11-01T04:16:17.843 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.844 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.844 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:17.844 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.196944+0000", 2023-11-01T04:16:17.844 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3300000000000003e-07 2023-11-01T04:16:17.845 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.845 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.845 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:17.845 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.197700+0000", 2023-11-01T04:16:17.845 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00075622600000000003 2023-11-01T04:16:17.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.846 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.846 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:17.846 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.197728+0000", 2023-11-01T04:16:17.846 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7991999999999999e-05 2023-11-01T04:16:17.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.847 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.847 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.847 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.197744+0000", 2023-11-01T04:16:17.847 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5109000000000001e-05 2023-11-01T04:16:17.848 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.848 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.848 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:17.848 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.197909+0000", 2023-11-01T04:16:17.848 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000165217 2023-11-01T04:16:17.849 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.849 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.849 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:17.849 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.197937+0000", 2023-11-01T04:16:17.849 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8717999999999998e-05 2023-11-01T04:16:17.850 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.850 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.850 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:17.850 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.299866+0000", 2023-11-01T04:16:17.850 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10192886399999999 2023-11-01T04:16:17.851 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.851 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.851 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:17.851 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.299936+0000", 2023-11-01T04:16:17.851 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9766999999999998e-05 2023-11-01T04:16:17.852 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.852 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:17.852 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.852 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.852 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.853 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.as2 36/31 ECSubWrite(tid=24, reqid=client.4559.0:140, at_version=36'4, trim_to=0'0, roll_forward_to=34'2))", 2023-11-01T04:16:17.853 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.198767+0000", 2023-11-01T04:16:17.853 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.6265800380000002, 2023-11-01T04:16:17.853 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101958549, 2023-11-01T04:16:17.853 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:17.854 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.854 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:17.854 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.854 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:17.854 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.198767+0000", 2023-11-01T04:16:17.855 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.855 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.855 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.855 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:17.855 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.198767+0000", 2023-11-01T04:16:17.856 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.856 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.856 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.856 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:17.856 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.198768+0000", 2023-11-01T04:16:17.857 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4169999999999999e-06 2023-11-01T04:16:17.857 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.857 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.857 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:17.857 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.198797+0000", 2023-11-01T04:16:17.857 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8928000000000002e-05 2023-11-01T04:16:17.858 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.858 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.858 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:17.858 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.198831+0000", 2023-11-01T04:16:17.858 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3915999999999999e-05 2023-11-01T04:16:17.859 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.859 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.859 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.859 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.198844+0000", 2023-11-01T04:16:17.859 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3106e-05 2023-11-01T04:16:17.859 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.860 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.860 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:17.860 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.299345+0000", 2023-11-01T04:16:17.860 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100500988 2023-11-01T04:16:17.860 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.861 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.861 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:17.861 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.299375+0000", 2023-11-01T04:16:17.861 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9465e-05 2023-11-01T04:16:17.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.862 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.862 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:17.862 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.300666+0000", 2023-11-01T04:16:17.862 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001291031 2023-11-01T04:16:17.862 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.862 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.863 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:17.863 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.300726+0000", 2023-11-01T04:16:17.863 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9697999999999997e-05 2023-11-01T04:16:17.863 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.864 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:17.864 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.864 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.864 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/31 ECSubWrite(tid=18, reqid=client.4559.0:125, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:17.864 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.281930+0000", 2023-11-01T04:16:17.865 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.5434168980000003, 2023-11-01T04:16:17.865 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20154303100000001, 2023-11-01T04:16:17.865 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:17.865 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.865 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:17.866 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.866 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:17.866 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281930+0000", 2023-11-01T04:16:17.866 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.866 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.867 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.867 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:17.867 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281930+0000", 2023-11-01T04:16:17.867 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.868 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.868 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.868 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:17.868 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281931+0000", 2023-11-01T04:16:17.868 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6400000000000001e-07 2023-11-01T04:16:17.869 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.869 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.869 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:17.869 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281961+0000", 2023-11-01T04:16:17.869 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9621999999999999e-05 2023-11-01T04:16:17.870 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.870 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.870 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:17.870 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281987+0000", 2023-11-01T04:16:17.870 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6316e-05 2023-11-01T04:16:17.871 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.871 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.871 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382103+0000", 2023-11-01T04:16:17.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100116649 2023-11-01T04:16:17.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.872 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:17.872 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.482335+0000", 2023-11-01T04:16:17.872 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100231813 2023-11-01T04:16:17.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.873 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.873 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:17.873 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.482364+0000", 2023-11-01T04:16:17.873 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8829000000000001e-05 2023-11-01T04:16:17.873 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.874 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.874 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:17.874 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.483427+0000", 2023-11-01T04:16:17.874 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010627970000000001 2023-11-01T04:16:17.874 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.874 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.875 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:17.875 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.483473+0000", 2023-11-01T04:16:17.875 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6241e-05 2023-11-01T04:16:17.875 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.875 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:17.876 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.876 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/31 ECSubWrite(tid=4, reqid=client.4559.0:136, at_version=36'3, trim_to=0'0, roll_forward_to=33'2))", 2023-11-01T04:16:17.876 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.284706+0000", 2023-11-01T04:16:17.876 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.54064099, 2023-11-01T04:16:17.877 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202677317, 2023-11-01T04:16:17.877 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:17.877 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.877 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:17.877 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.878 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:17.878 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.284706+0000", 2023-11-01T04:16:17.878 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.878 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.878 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.878 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:17.879 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.284706+0000", 2023-11-01T04:16:17.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.879 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.879 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.879 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:17.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.284707+0000", 2023-11-01T04:16:17.880 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8599999999999998e-07 2023-11-01T04:16:17.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.880 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:17.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.285485+0000", 2023-11-01T04:16:17.881 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077885900000000002 2023-11-01T04:16:17.881 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.881 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.881 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:17.881 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.285511+0000", 2023-11-01T04:16:17.881 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5378999999999999e-05 2023-11-01T04:16:17.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.882 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.882 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.882 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385600+0000", 2023-11-01T04:16:17.882 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100089631 2023-11-01T04:16:17.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.883 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.883 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:17.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385749+0000", 2023-11-01T04:16:17.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014876399999999999 2023-11-01T04:16:17.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.884 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.884 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:17.884 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385771+0000", 2023-11-01T04:16:17.884 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1535e-05 2023-11-01T04:16:17.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.884 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.885 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:17.885 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.487315+0000", 2023-11-01T04:16:17.885 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101544735 2023-11-01T04:16:17.885 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.885 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.886 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:17.886 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.487383+0000", 2023-11-01T04:16:17.886 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7928000000000004e-05 2023-11-01T04:16:17.886 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.886 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:17.887 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.887 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.887 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 36/31 ECSubWrite(tid=14, reqid=client.4559.0:146, at_version=36'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:17.887 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.322569+0000", 2023-11-01T04:16:17.887 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.5027780960000001, 2023-11-01T04:16:17.888 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20180657299999999, 2023-11-01T04:16:17.888 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:17.888 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.888 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:17.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:17.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322569+0000", 2023-11-01T04:16:17.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.889 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:17.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322569+0000", 2023-11-01T04:16:17.890 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.890 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.890 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:17.891 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322571+0000", 2023-11-01T04:16:17.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8029999999999999e-06 2023-11-01T04:16:17.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.891 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.891 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:17.891 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322759+0000", 2023-11-01T04:16:17.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018862300000000001 2023-11-01T04:16:17.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:17.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322790+0000", 2023-11-01T04:16:17.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0712e-05 2023-11-01T04:16:17.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322807+0000", 2023-11-01T04:16:17.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6566e-05 2023-11-01T04:16:17.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:17.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.423031+0000", 2023-11-01T04:16:17.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10022463600000001 2023-11-01T04:16:17.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:17.895 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.423046+0000", 2023-11-01T04:16:17.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4632e-05 2023-11-01T04:16:17.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:17.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524322+0000", 2023-11-01T04:16:17.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101275743 2023-11-01T04:16:17.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:17.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524376+0000", 2023-11-01T04:16:17.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3857999999999998e-05 2023-11-01T04:16:17.897 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.897 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:17.897 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.898 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpRead(3.4s3 36/31 ECSubRead(tid=19, to_read={3:22b97aa6:::smithi19039307-42:head=0,4096,0}, subchunks={3:22b97aa6:::smithi19039307-42:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:17.898 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.382157+0000", 2023-11-01T04:16:17.898 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.443190059, 2023-11-01T04:16:17.898 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101265231, 2023-11-01T04:16:17.899 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:17.899 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.899 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:17.899 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.899 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:17.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382157+0000", 2023-11-01T04:16:17.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:17.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382157+0000", 2023-11-01T04:16:17.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:17.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382158+0000", 2023-11-01T04:16:17.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3939999999999999e-06 2023-11-01T04:16:17.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:17.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382179+0000", 2023-11-01T04:16:17.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0588999999999999e-05 2023-11-01T04:16:17.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:17.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382214+0000", 2023-11-01T04:16:17.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5318999999999998e-05 2023-11-01T04:16:17.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382246+0000", 2023-11-01T04:16:17.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1228e-05 2023-11-01T04:16:17.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:17.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.482792+0000", 2023-11-01T04:16:17.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10054644 2023-11-01T04:16:17.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:17.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.483422+0000", 2023-11-01T04:16:17.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00063026100000000002 2023-11-01T04:16:17.906 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.906 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:17.907 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.907 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 36/31 ECSubWrite(tid=16, reqid=client.4559.0:147, at_version=36'10, trim_to=0'0, roll_forward_to=36'8))", 2023-11-01T04:16:17.907 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.382654+0000", 2023-11-01T04:16:17.907 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.4426934949999999, 2023-11-01T04:16:17.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14175697400000001, 2023-11-01T04:16:17.908 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:17.908 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.908 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:17.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:17.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382654+0000", 2023-11-01T04:16:17.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:17.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382654+0000", 2023-11-01T04:16:17.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:17.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382655+0000", 2023-11-01T04:16:17.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.933e-06 2023-11-01T04:16:17.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:17.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382823+0000", 2023-11-01T04:16:17.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000167806 2023-11-01T04:16:17.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:17.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382856+0000", 2023-11-01T04:16:17.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2255999999999997e-05 2023-11-01T04:16:17.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382926+0000", 2023-11-01T04:16:17.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0759999999999993e-05 2023-11-01T04:16:17.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:17.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.424403+0000", 2023-11-01T04:16:17.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.041477075000000002 2023-11-01T04:16:17.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:17.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.424416+0000", 2023-11-01T04:16:17.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2249000000000001e-05 2023-11-01T04:16:17.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:17.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524381+0000", 2023-11-01T04:16:17.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.099965719999999994 2023-11-01T04:16:17.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:17.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524411+0000", 2023-11-01T04:16:17.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9175000000000001e-05 2023-11-01T04:16:17.917 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.917 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:17.917 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.918 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 36/31 ECSubWrite(tid=17, reqid=client.4559.0:148, at_version=36'11, trim_to=0'0, roll_forward_to=36'8))", 2023-11-01T04:16:17.918 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.385040+0000", 2023-11-01T04:16:17.918 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.4403070319999998, 2023-11-01T04:16:17.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14044638700000001, 2023-11-01T04:16:17.919 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:17.919 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.919 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:17.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:17.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385040+0000", 2023-11-01T04:16:17.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:17.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385040+0000", 2023-11-01T04:16:17.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:17.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385042+0000", 2023-11-01T04:16:17.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7129999999999999e-06 2023-11-01T04:16:17.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:17.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385087+0000", 2023-11-01T04:16:17.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4895000000000003e-05 2023-11-01T04:16:17.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:17.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385121+0000", 2023-11-01T04:16:17.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4180999999999997e-05 2023-11-01T04:16:17.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385145+0000", 2023-11-01T04:16:17.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3949000000000001e-05 2023-11-01T04:16:17.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:17.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.523994+0000", 2023-11-01T04:16:17.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13884963 2023-11-01T04:16:17.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:17.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524008+0000", 2023-11-01T04:16:17.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3719e-05 2023-11-01T04:16:17.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:17.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.525407+0000", 2023-11-01T04:16:17.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013987229999999999 2023-11-01T04:16:17.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:17.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.525486+0000", 2023-11-01T04:16:17.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9577000000000001e-05 2023-11-01T04:16:17.928 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.928 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:17.928 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.929 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/31 ECSubWrite(tid=5, reqid=client.4559.0:137, at_version=36'4, trim_to=0'0, roll_forward_to=33'2))", 2023-11-01T04:16:17.929 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.385648+0000", 2023-11-01T04:16:17.929 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.4396993440000001, 2023-11-01T04:16:17.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102787935, 2023-11-01T04:16:17.929 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:17.930 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.930 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:17.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:17.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385648+0000", 2023-11-01T04:16:17.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:17.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385648+0000", 2023-11-01T04:16:17.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:17.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385649+0000", 2023-11-01T04:16:17.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1170000000000001e-06 2023-11-01T04:16:17.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:17.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385676+0000", 2023-11-01T04:16:17.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7209e-05 2023-11-01T04:16:17.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:17.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385694+0000", 2023-11-01T04:16:17.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8014999999999999e-05 2023-11-01T04:16:17.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385707+0000", 2023-11-01T04:16:17.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3332999999999999e-05 2023-11-01T04:16:17.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:17.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.486918+0000", 2023-11-01T04:16:17.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101210883 2023-11-01T04:16:17.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:17.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.486940+0000", 2023-11-01T04:16:17.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1594999999999999e-05 2023-11-01T04:16:17.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:17.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.488346+0000", 2023-11-01T04:16:17.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014057729999999999 2023-11-01T04:16:17.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:17.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.488436+0000", 2023-11-01T04:16:17.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.001e-05 2023-11-01T04:16:17.938 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.939 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:17.939 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.939 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.6s2 36/31 ECSubWrite(tid=22, reqid=client.4559.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=36'4))", 2023-11-01T04:16:17.939 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.688041+0000", 2023-11-01T04:16:17.940 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.1373063079999999, 2023-11-01T04:16:17.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102320034, 2023-11-01T04:16:17.940 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:17.940 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.940 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:17.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:17.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688041+0000", 2023-11-01T04:16:17.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:17.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688041+0000", 2023-11-01T04:16:17.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:17.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688042+0000", 2023-11-01T04:16:17.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.677e-06 2023-11-01T04:16:17.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:17.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688063+0000", 2023-11-01T04:16:17.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1078999999999999e-05 2023-11-01T04:16:17.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:17.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688094+0000", 2023-11-01T04:16:17.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0839000000000001e-05 2023-11-01T04:16:17.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688111+0000", 2023-11-01T04:16:17.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6492999999999999e-05 2023-11-01T04:16:17.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:17.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.789077+0000", 2023-11-01T04:16:17.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10096656499999999 2023-11-01T04:16:17.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:17.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.789093+0000", 2023-11-01T04:16:17.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6028999999999998e-05 2023-11-01T04:16:17.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:17.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.790298+0000", 2023-11-01T04:16:17.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001204917 2023-11-01T04:16:17.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:17.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.790361+0000", 2023-11-01T04:16:17.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2434999999999994e-05 2023-11-01T04:16:17.949 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.949 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:17.949 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.950 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.2s5 36/31 ECSubWrite(tid=197, reqid=client.4559.0:160, at_version=36'16, trim_to=0'0, roll_forward_to=36'15))", 2023-11-01T04:16:17.950 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.113843+0000", 2023-11-01T04:16:17.950 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.7115038629999999, 2023-11-01T04:16:17.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101971642, 2023-11-01T04:16:17.950 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:17.951 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.951 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:17.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:17.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.113843+0000", 2023-11-01T04:16:17.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:17.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.113843+0000", 2023-11-01T04:16:17.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:17.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.113845+0000", 2023-11-01T04:16:17.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0899999999999999e-06 2023-11-01T04:16:17.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:17.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.113897+0000", 2023-11-01T04:16:17.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2061999999999997e-05 2023-11-01T04:16:17.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:17.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.113938+0000", 2023-11-01T04:16:17.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0887999999999999e-05 2023-11-01T04:16:17.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.113962+0000", 2023-11-01T04:16:17.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3502999999999998e-05 2023-11-01T04:16:17.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:17.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.214220+0000", 2023-11-01T04:16:17.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100257795 2023-11-01T04:16:17.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:17.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.214239+0000", 2023-11-01T04:16:17.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9202999999999999e-05 2023-11-01T04:16:17.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:17.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.215732+0000", 2023-11-01T04:16:17.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001493264 2023-11-01T04:16:17.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:17.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.215815+0000", 2023-11-01T04:16:17.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2837000000000007e-05 2023-11-01T04:16:17.960 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.960 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:17.960 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.960 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "pg_update_log_missing(3.2s5 epoch 36/31 rep_tid 199 entries 36'17 (0'0) error 3:4b7227f5:::smithi19039307-29:head by client.4559.0:171 0.000000 -2 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 36'15)", 2023-11-01T04:16:17.961 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.319882+0000", 2023-11-01T04:16:17.961 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.505464607, 2023-11-01T04:16:17.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10195180700000001, 2023-11-01T04:16:17.961 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:17.961 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.961 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:17.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:17.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.319882+0000", 2023-11-01T04:16:17.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:17.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.319882+0000", 2023-11-01T04:16:17.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:17.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.319885+0000", 2023-11-01T04:16:17.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.384e-06 2023-11-01T04:16:17.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:17.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.319906+0000", 2023-11-01T04:16:17.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0798999999999999e-05 2023-11-01T04:16:17.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:17.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.319994+0000", 2023-11-01T04:16:17.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8713000000000003e-05 2023-11-01T04:16:17.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.320037+0000", 2023-11-01T04:16:17.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2898000000000002e-05 2023-11-01T04:16:17.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:17.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.420364+0000", 2023-11-01T04:16:17.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100326319 2023-11-01T04:16:17.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:17.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.421834+0000", 2023-11-01T04:16:17.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001470694 2023-11-01T04:16:17.968 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.968 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:17.968 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.969 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 37/31 ECSubWrite(tid=21, reqid=client.4559.0:186, at_version=37'14, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:16:17.969 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.967688+0000", 2023-11-01T04:16:17.969 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.85765856100000004, 2023-11-01T04:16:17.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102405465, 2023-11-01T04:16:17.970 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:17.970 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.970 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:17.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:17.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967688+0000", 2023-11-01T04:16:17.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:17.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967688+0000", 2023-11-01T04:16:17.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:17.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967690+0000", 2023-11-01T04:16:17.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.657e-06 2023-11-01T04:16:17.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:17.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967739+0000", 2023-11-01T04:16:17.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8513999999999997e-05 2023-11-01T04:16:17.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:17.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967769+0000", 2023-11-01T04:16:17.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0852999999999998e-05 2023-11-01T04:16:17.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.067914+0000", 2023-11-01T04:16:17.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100144572 2023-11-01T04:16:17.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:17.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.068061+0000", 2023-11-01T04:16:17.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014692700000000001 2023-11-01T04:16:17.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:17.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.068075+0000", 2023-11-01T04:16:17.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4279e-05 2023-11-01T04:16:17.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:17.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.069999+0000", 2023-11-01T04:16:17.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019239940000000001 2023-11-01T04:16:17.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:17.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.070094+0000", 2023-11-01T04:16:17.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4668999999999998e-05 2023-11-01T04:16:17.979 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.979 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:17.979 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.979 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 37/31 ECSubWrite(tid=22, reqid=client.4559.0:197, at_version=37'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:17.980 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:16:17.980 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.78507548400000005, 2023-11-01T04:16:17.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.301677274, 2023-11-01T04:16:17.980 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:17.980 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.980 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:17.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:17.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:16:17.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:17.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:16:17.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:17.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040273+0000", 2023-11-01T04:16:17.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3370000000000001e-06 2023-11-01T04:16:17.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:17.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040307+0000", 2023-11-01T04:16:17.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3714999999999998e-05 2023-11-01T04:16:17.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:17.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040331+0000", 2023-11-01T04:16:17.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4409e-05 2023-11-01T04:16:17.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140479+0000", 2023-11-01T04:16:17.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10014764900000001 2023-11-01T04:16:17.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:17.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.240734+0000", 2023-11-01T04:16:17.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100255645 2023-11-01T04:16:17.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:17.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.240751+0000", 2023-11-01T04:16:17.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6852000000000001e-05 2023-11-01T04:16:17.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:17.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341894+0000", 2023-11-01T04:16:17.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101143302 2023-11-01T04:16:17.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:17.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341949+0000", 2023-11-01T04:16:17.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4364999999999999e-05 2023-11-01T04:16:17.990 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.990 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:17.990 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:17.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.991 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 37/31 ECSubWrite(tid=23, reqid=client.4559.0:198, at_version=37'10, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:17.991 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.140534+0000", 2023-11-01T04:16:17.991 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.68481259000000005, 2023-11-01T04:16:17.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202392451, 2023-11-01T04:16:17.992 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:17.992 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:17.992 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:17.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:17.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140534+0000", 2023-11-01T04:16:17.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:17.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140534+0000", 2023-11-01T04:16:17.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:17.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:17.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140536+0000", 2023-11-01T04:16:17.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.22e-06 2023-11-01T04:16:17.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:17.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140578+0000", 2023-11-01T04:16:17.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2715e-05 2023-11-01T04:16:17.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:17.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140603+0000", 2023-11-01T04:16:17.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4609000000000001e-05 2023-11-01T04:16:17.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:17.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140616+0000", 2023-11-01T04:16:17.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2763e-05 2023-11-01T04:16:17.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:17.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341493+0000", 2023-11-01T04:16:17.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.200877214 2023-11-01T04:16:17.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:17.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:17.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:17.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341508+0000", 2023-11-01T04:16:17.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5013e-05 2023-11-01T04:16:17.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:18.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.342869+0000", 2023-11-01T04:16:18.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001361183 2023-11-01T04:16:18.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:18.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.342927+0000", 2023-11-01T04:16:18.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7734e-05 2023-11-01T04:16:18.001 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:18.001 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:18.002 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:18.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.002 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 37/31 ECSubWrite(tid=13, reqid=client.4559.0:181, at_version=37'8, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:18.002 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.248764+0000", 2023-11-01T04:16:18.002 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.57658320500000004, 2023-11-01T04:16:18.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20321630399999999, 2023-11-01T04:16:18.003 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:18.003 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:18.003 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:18.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:18.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.248764+0000", 2023-11-01T04:16:18.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:18.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:18.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.248764+0000", 2023-11-01T04:16:18.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:18.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:18.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.248766+0000", 2023-11-01T04:16:18.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8300000000000001e-06 2023-11-01T04:16:18.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:18.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.249255+0000", 2023-11-01T04:16:18.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00048892399999999996 2023-11-01T04:16:18.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:18.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.249286+0000", 2023-11-01T04:16:18.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1004999999999999e-05 2023-11-01T04:16:18.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:18.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.249302+0000", 2023-11-01T04:16:18.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6169999999999999e-05 2023-11-01T04:16:18.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:18.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.349516+0000", 2023-11-01T04:16:18.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10021408599999999 2023-11-01T04:16:18.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:18.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.349530+0000", 2023-11-01T04:16:18.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4472e-05 2023-11-01T04:16:18.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:18.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451931+0000", 2023-11-01T04:16:18.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10240034100000001 2023-11-01T04:16:18.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:18.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451980+0000", 2023-11-01T04:16:18.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9475999999999999e-05 2023-11-01T04:16:18.012 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:18.012 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:18.012 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:18.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.012 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 37/31 ECSubWrite(tid=14, reqid=client.4559.0:182, at_version=37'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:18.013 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.253221+0000", 2023-11-01T04:16:18.013 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.57212604899999997, 2023-11-01T04:16:18.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19911255999999999, 2023-11-01T04:16:18.013 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:18.013 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:18.014 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:18.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:18.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253221+0000", 2023-11-01T04:16:18.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:18.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:18.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253221+0000", 2023-11-01T04:16:18.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:18.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:18.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253222+0000", 2023-11-01T04:16:18.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1570000000000001e-06 2023-11-01T04:16:18.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:18.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253418+0000", 2023-11-01T04:16:18.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019597500000000001 2023-11-01T04:16:18.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:18.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253443+0000", 2023-11-01T04:16:18.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5055999999999999e-05 2023-11-01T04:16:18.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:18.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253459+0000", 2023-11-01T04:16:18.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5495999999999999e-05 2023-11-01T04:16:18.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:18.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451029+0000", 2023-11-01T04:16:18.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19757031799999999 2023-11-01T04:16:18.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:18.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451039+0000", 2023-11-01T04:16:18.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0062e-05 2023-11-01T04:16:18.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:18.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.452298+0000", 2023-11-01T04:16:18.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012584849999999999 2023-11-01T04:16:18.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:18.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.452334+0000", 2023-11-01T04:16:18.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6010999999999998e-05 2023-11-01T04:16:18.023 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:18.023 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:18.023 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:18.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.024 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.2s5 37/31 ECSubWrite(tid=206, reqid=client.4559.0:176, at_version=37'19, trim_to=0'0, roll_forward_to=36'16))", 2023-11-01T04:16:18.024 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.253930+0000", 2023-11-01T04:16:18.024 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.57141693299999996, 2023-11-01T04:16:18.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10315031600000001, 2023-11-01T04:16:18.025 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:18.025 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:18.025 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:18.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:18.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253930+0000", 2023-11-01T04:16:18.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:18.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:18.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253930+0000", 2023-11-01T04:16:18.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:18.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:18.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253930+0000", 2023-11-01T04:16:18.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5999999999999999e-07 2023-11-01T04:16:18.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:18.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.254197+0000", 2023-11-01T04:16:18.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026679500000000001 2023-11-01T04:16:18.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:18.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.254222+0000", 2023-11-01T04:16:18.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4453000000000001e-05 2023-11-01T04:16:18.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:18.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.254237+0000", 2023-11-01T04:16:18.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4946000000000001e-05 2023-11-01T04:16:18.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:18.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.354451+0000", 2023-11-01T04:16:18.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10021408599999999 2023-11-01T04:16:18.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:18.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.354465+0000", 2023-11-01T04:16:18.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4549000000000001e-05 2023-11-01T04:16:18.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:18.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.357011+0000", 2023-11-01T04:16:18.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025457520000000001 2023-11-01T04:16:18.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:18.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.357080+0000", 2023-11-01T04:16:18.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9375000000000006e-05 2023-11-01T04:16:18.034 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:18.034 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:18.034 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:18.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.035 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 37/31 ECSubWrite(tid=15, reqid=client.4559.0:183, at_version=37'10, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:18.035 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.255859+0000", 2023-11-01T04:16:18.035 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.569488357, 2023-11-01T04:16:18.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19868693900000001, 2023-11-01T04:16:18.035 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:18.035 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:18.036 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:18.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:18.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255859+0000", 2023-11-01T04:16:18.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:18.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:18.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255859+0000", 2023-11-01T04:16:18.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:18.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:18.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255860+0000", 2023-11-01T04:16:18.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2300000000000001e-06 2023-11-01T04:16:18.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:18.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255894+0000", 2023-11-01T04:16:18.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3796e-05 2023-11-01T04:16:18.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:18.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255923+0000", 2023-11-01T04:16:18.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9352e-05 2023-11-01T04:16:18.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:18.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255936+0000", 2023-11-01T04:16:18.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3232e-05 2023-11-01T04:16:18.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:18.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451650+0000", 2023-11-01T04:16:18.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19571368 2023-11-01T04:16:18.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:18.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451665+0000", 2023-11-01T04:16:18.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5119e-05 2023-11-01T04:16:18.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:18.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.454503+0000", 2023-11-01T04:16:18.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028377400000000001 2023-11-01T04:16:18.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:18.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:18.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:18.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.454546+0000", 2023-11-01T04:16:18.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2790000000000002e-05 2023-11-01T04:16:18.044 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:18.044 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:18.044 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:18.044 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:18.045 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:18.045 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:18.046 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:18.046 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-11-01T04:16:18.047 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:18.048 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-01T04:16:18.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-01T04:16:18.049 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:18.046+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:18.049 INFO:tasks.rados.rados.0.smithi190.stdout:75: finishing copy_from to smithi19039307-49 2023-11-01T04:16:18.049 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 49 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2023-11-01T04:16:18.049 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2023-11-01T04:16:18.049 INFO:tasks.rados.rados.0.smithi190.stdout:62: done (15 left) 2023-11-01T04:16:18.050 INFO:tasks.rados.rados.0.smithi190.stdout:64: done (14 left) 2023-11-01T04:16:18.050 INFO:tasks.rados.rados.0.smithi190.stdout:65: done (13 left) 2023-11-01T04:16:18.050 INFO:tasks.rados.rados.0.smithi190.stdout:66: done (12 left) 2023-11-01T04:16:18.050 INFO:tasks.rados.rados.0.smithi190.stdout:67: done (11 left) 2023-11-01T04:16:18.050 INFO:tasks.rados.rados.0.smithi190.stdout:68: done (10 left) 2023-11-01T04:16:18.051 INFO:tasks.rados.rados.0.smithi190.stdout:69: done (9 left) 2023-11-01T04:16:18.051 INFO:tasks.rados.rados.0.smithi190.stdout:71: done (8 left) 2023-11-01T04:16:18.051 INFO:tasks.rados.rados.0.smithi190.stdout:72: done (7 left) 2023-11-01T04:16:18.051 INFO:tasks.rados.rados.0.smithi190.stdout:73: done (6 left) 2023-11-01T04:16:18.051 INFO:tasks.rados.rados.0.smithi190.stdout:75: done (5 left) 2023-11-01T04:16:18.052 INFO:tasks.rados.rados.0.smithi190.stdout:76: done (4 left) 2023-11-01T04:16:18.052 INFO:tasks.rados.rados.0.smithi190.stdout:77: done (3 left) 2023-11-01T04:16:18.052 INFO:tasks.rados.rados.0.smithi190.stdout:78: done (2 left) 2023-11-01T04:16:18.052 INFO:tasks.rados.rados.0.smithi190.stdout:79: done (1 left) 2023-11-01T04:16:18.052 INFO:tasks.rados.rados.0.smithi190.stdout:80: done (0 left) 2023-11-01T04:16:18.052 INFO:tasks.rados.rados.0.smithi190.stdout:81: rollback oid 40 current snap is 2 2023-11-01T04:16:18.052 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 40 to 1 2023-11-01T04:16:18.053 INFO:tasks.rados.rados.0.smithi190.stdout:82: snap_create 2023-11-01T04:16:18.053 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:18.053 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-01T04:16:18.054 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:18.054 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-01T04:16:18.054 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:18.055 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-01T04:16:18.055 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:18.056 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-01T04:16:18.056 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:18.056 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-01T04:16:18.057 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:18.057 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-01T04:16:18.057 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:18.058 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-01T04:16:18.058 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:18.058 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-01T04:16:18.061 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:18.062 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-01T04:16:18.083 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:18.083 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-01T04:16:18.101 INFO:teuthology.orchestra.run.smithi153.stdout:osd.12: {} 2023-11-01T04:16:18.102 INFO:teuthology.orchestra.run.smithi153.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-01T04:16:18.122 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:18.122 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-01T04:16:18.147 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:18.146+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:18.224 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:18.225 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:18.240 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-01T04:16:18.248 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:18.248+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:18.348 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:18.346+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:18.364 INFO:tasks.rados.rados.0.smithi190.stdout:81: finishing rollback tid 0 to smithi19039307-40 2023-11-01T04:16:18.364 INFO:tasks.rados.rados.0.smithi190.stdout:81: finishing rollback tid 1 to smithi19039307-40 2023-11-01T04:16:18.386 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:18.386 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-01T04:16:18.400 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:18.400 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:18.413 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:18.413 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-01T04:16:18.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-01T04:16:18.457 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:18.458+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:18.464 INFO:tasks.rados.rados.0.smithi190.stdout:81: finishing rollback tid 2 to smithi19039307-40 2023-11-01T04:16:18.464 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2023-11-01T04:16:18.550 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:18.549+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:18.578 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:18.578 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:18.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-01T04:16:18.651 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:18.648+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:18.708 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:18.709 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:18.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-01T04:16:18.752 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:18.752+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:18.839 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:18.839 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:18.853 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-01T04:16:18.854 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:18.852+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:18.950 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2023-11-01T04:16:18.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-11-01T04:16:18.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-11-01T04:16:18.953 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:18.949+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:18.957 INFO:tasks.rados.rados.0.smithi190.stdout:81: done (1 left) 2023-11-01T04:16:18.957 INFO:tasks.rados.rados.0.smithi190.stdout:82: done (0 left) 2023-11-01T04:16:18.957 INFO:tasks.rados.rados.0.smithi190.stdout:83: rollback oid 9 current snap is 3 2023-11-01T04:16:18.957 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 9 to 0 2023-11-01T04:16:18.957 INFO:tasks.rados.rados.0.smithi190.stdout:84: read oid 20 snap 1 2023-11-01T04:16:18.958 INFO:tasks.rados.rados.0.smithi190.stdout:84: expect (ObjNum 29 snap 0 seq_num 29) 2023-11-01T04:16:18.958 INFO:tasks.rados.rados.0.smithi190.stdout:85: read oid 41 snap -1 2023-11-01T04:16:18.958 INFO:tasks.rados.rados.0.smithi190.stdout:85: expect (ObjNum 40 snap 0 seq_num 40) 2023-11-01T04:16:18.958 INFO:tasks.rados.rados.0.smithi190.stdout:86: read oid 5 snap -1 2023-11-01T04:16:18.958 INFO:tasks.rados.rados.0.smithi190.stdout:86: expect (ObjNum 4 snap 0 seq_num 4) 2023-11-01T04:16:18.958 INFO:tasks.rados.rados.0.smithi190.stdout:87: read oid 4 snap -1 2023-11-01T04:16:18.959 INFO:tasks.rados.rados.0.smithi190.stdout:87: expect (ObjNum 3 snap 0 seq_num 3) 2023-11-01T04:16:18.959 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 3 current snap is 3 2023-11-01T04:16:18.959 INFO:tasks.rados.rados.0.smithi190.stdout:88: seq_num 54 ranges {49152=540672,589824=376832} 2023-11-01T04:16:18.966 INFO:tasks.rados.rados.0.smithi190.stdout:88: writing smithi19039307-3 from 49152 to 589824 tid 1 2023-11-01T04:16:18.972 INFO:tasks.rados.rados.0.smithi190.stdout:88: writing smithi19039307-3 from 589824 to 966656 tid 2 2023-11-01T04:16:18.973 INFO:tasks.rados.rados.0.smithi190.stdout:89: read oid 27 snap -1 2023-11-01T04:16:18.973 INFO:tasks.rados.rados.0.smithi190.stdout:89: expect (ObjNum 26 snap 0 seq_num 26) 2023-11-01T04:16:18.973 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 17 current snap is 3 2023-11-01T04:16:18.973 INFO:tasks.rados.rados.0.smithi190.stdout:90: seq_num 55 ranges {278528=557056,835584=376832} 2023-11-01T04:16:18.974 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:18.974 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:18.986 INFO:tasks.rados.rados.0.smithi190.stdout:90: writing smithi19039307-17 from 278528 to 835584 tid 1 2023-11-01T04:16:18.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-01T04:16:18.993 INFO:tasks.rados.rados.0.smithi190.stdout:90: writing smithi19039307-17 from 835584 to 1212416 tid 2 2023-11-01T04:16:18.993 INFO:tasks.rados.rados.0.smithi190.stdout:91: rollback oid 36 current snap is 3 2023-11-01T04:16:18.993 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 36 to 2 2023-11-01T04:16:18.994 INFO:tasks.rados.rados.0.smithi190.stdout:92: rollback oid 7 current snap is 3 2023-11-01T04:16:18.994 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 7 to 0 2023-11-01T04:16:18.994 INFO:tasks.rados.rados.0.smithi190.stdout:93: snap_create 2023-11-01T04:16:19.053 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:19.049+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:19.109 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:19.109 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:19.123 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-01T04:16:19.169 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:19.170+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:19.239 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:19.239 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:19.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-01T04:16:19.255 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:19.252+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:19.355 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:19.349+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:19.356 INFO:tasks.rados.rados.0.smithi190.stdout:88: finishing write tid 1 to smithi19039307-3 2023-11-01T04:16:19.359 INFO:tasks.rados.rados.0.smithi190.stdout:88: finishing write tid 2 to smithi19039307-3 2023-11-01T04:16:19.370 INFO:tasks.rados.rados.0.smithi190.stdout:83: finishing rollback tid 0 to smithi19039307-9 2023-11-01T04:16:19.377 INFO:tasks.rados.rados.0.smithi190.stdout:85: expect (ObjNum 40 snap 0 seq_num 40) 2023-11-01T04:16:19.402 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:19.402 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:19.415 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-01T04:16:19.428 INFO:tasks.rados.rados.0.smithi190.stdout:89: expect (ObjNum 26 snap 0 seq_num 26) 2023-11-01T04:16:19.451 INFO:tasks.rados.rados.0.smithi190.stdout:90: finishing write tid 1 to smithi19039307-17 2023-11-01T04:16:19.451 INFO:tasks.rados.rados.0.smithi190.stdout:91: finishing rollback tid 0 to smithi19039307-36 2023-11-01T04:16:19.451 INFO:tasks.rados.rados.0.smithi190.stdout:91: finishing rollback tid 1 to smithi19039307-36 2023-11-01T04:16:19.451 INFO:tasks.rados.rados.0.smithi190.stdout:91: finishing rollback tid 2 to smithi19039307-36 2023-11-01T04:16:19.451 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 36 v 23 (ObjNum 35 snap 0 seq_num 35) dirty exists 2023-11-01T04:16:19.456 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:19.455+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:19.458 INFO:tasks.rados.rados.0.smithi190.stdout:88: finishing write tid 3 to smithi19039307-3 2023-11-01T04:16:19.496 INFO:tasks.rados.rados.0.smithi190.stdout:92: finishing rollback tid 0 to smithi19039307-7 2023-11-01T04:16:19.497 INFO:tasks.rados.rados.0.smithi190.stdout:92: finishing rollback tid 1 to smithi19039307-7 2023-11-01T04:16:19.497 INFO:tasks.rados.rados.0.smithi190.stdout:92: finishing rollback tid 2 to smithi19039307-7 2023-11-01T04:16:19.497 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 7 v 14 (ObjNum 6 snap 0 seq_num 6) dirty exists 2023-11-01T04:16:19.546 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:19.547 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:19.557 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:19.558+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:19.560 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-01T04:16:19.561 INFO:tasks.rados.rados.0.smithi190.stdout:88: finishing write tid 4 to smithi19039307-3 2023-11-01T04:16:19.561 INFO:tasks.rados.rados.0.smithi190.stdout:88: oid 3 updating version 0 to 10 2023-11-01T04:16:19.562 INFO:tasks.rados.rados.0.smithi190.stdout:88: oid 3 version 10 is already newer than 8 2023-11-01T04:16:19.562 INFO:tasks.rados.rados.0.smithi190.stdout:88: oid 3 version 10 is already newer than 9 2023-11-01T04:16:19.562 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 3 v 10 (ObjNum 54 snap 3 seq_num 54) dirty exists 2023-11-01T04:16:19.562 INFO:tasks.rados.rados.0.smithi190.stdout:88: left oid 3 (ObjNum 54 snap 3 seq_num 54) 2023-11-01T04:16:19.571 INFO:tasks.rados.rados.0.smithi190.stdout:83: finishing rollback tid 1 to smithi19039307-9 2023-11-01T04:16:19.571 INFO:tasks.rados.rados.0.smithi190.stdout:83: finishing rollback tid 2 to smithi19039307-9 2023-11-01T04:16:19.572 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 9 v 8 (ObjNum 8 snap 0 seq_num 8) dirty exists 2023-11-01T04:16:19.577 INFO:tasks.rados.rados.0.smithi190.stdout:87: expect (ObjNum 3 snap 0 seq_num 3) 2023-11-01T04:16:19.658 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:19.657+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:19.679 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:19.679 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:19.693 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-01T04:16:19.701 INFO:tasks.rados.rados.0.smithi190.stdout:84: expect (ObjNum 29 snap 0 seq_num 29) 2023-11-01T04:16:19.737 INFO:tasks.rados.rados.0.smithi190.stdout:90: finishing write tid 2 to smithi19039307-17 2023-11-01T04:16:19.759 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:19.753+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:19.765 INFO:tasks.rados.rados.0.smithi190.stdout:86: expect (ObjNum 4 snap 0 seq_num 4) 2023-11-01T04:16:19.803 INFO:tasks.rados.rados.0.smithi190.stdout:90: finishing write tid 3 to smithi19039307-17 2023-11-01T04:16:19.819 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:19.819 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:19.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-01T04:16:19.860 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:19.858+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:19.954 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:19.954 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:19.961 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:19.958+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:19.967 INFO:teuthology.orchestra.run.smithi153.stderr:noscrub is set 2023-11-01T04:16:19.967 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.0 to 1 (10000) 2023-11-01T04:16:19.971 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-01T04:16:19.974 INFO:tasks.rados.rados.0.smithi190.stdout:83: done (10 left) 2023-11-01T04:16:19.974 INFO:tasks.rados.rados.0.smithi190.stdout:84: done (9 left) 2023-11-01T04:16:19.974 INFO:tasks.rados.rados.0.smithi190.stdout:85: done (8 left) 2023-11-01T04:16:19.974 INFO:tasks.rados.rados.0.smithi190.stdout:86: done (7 left) 2023-11-01T04:16:19.975 INFO:tasks.rados.rados.0.smithi190.stdout:87: done (6 left) 2023-11-01T04:16:19.975 INFO:tasks.rados.rados.0.smithi190.stdout:88: done (5 left) 2023-11-01T04:16:19.975 INFO:tasks.rados.rados.0.smithi190.stdout:89: done (4 left) 2023-11-01T04:16:19.975 INFO:tasks.rados.rados.0.smithi190.stdout:91: done (3 left) 2023-11-01T04:16:19.975 INFO:tasks.rados.rados.0.smithi190.stdout:92: done (2 left) 2023-11-01T04:16:19.976 INFO:tasks.rados.rados.0.smithi190.stdout:93: done (1 left) 2023-11-01T04:16:19.976 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 43 current snap is 4 2023-11-01T04:16:19.976 INFO:tasks.rados.rados.0.smithi190.stdout:94: seq_num 56 ranges {720896=573440,1294336=376832} 2023-11-01T04:16:19.994 INFO:tasks.rados.rados.0.smithi190.stdout:94: writing smithi19039307-43 from 720896 to 1294336 tid 1 2023-11-01T04:16:20.000 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-11-01T04:16:20.001 INFO:tasks.rados.rados.0.smithi190.stdout:94: writing smithi19039307-43 from 1294336 to 1671168 tid 2 2023-11-01T04:16:20.001 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 19 current snap is 4 2023-11-01T04:16:20.002 INFO:tasks.rados.rados.0.smithi190.stdout:95: seq_num 57 ranges {311296=589824,901120=376832} 2023-11-01T04:16:20.015 INFO:tasks.rados.rados.0.smithi190.stdout:95: writing smithi19039307-19 from 311296 to 901120 tid 1 2023-11-01T04:16:20.022 INFO:tasks.rados.rados.0.smithi190.stdout:95: writing smithi19039307-19 from 901120 to 1277952 tid 2 2023-11-01T04:16:20.022 INFO:tasks.rados.rados.0.smithi190.stdout:96: read oid 20 snap -1 2023-11-01T04:16:20.023 INFO:tasks.rados.rados.0.smithi190.stdout:96: expect (ObjNum 29 snap 0 seq_num 29) 2023-11-01T04:16:20.023 INFO:tasks.rados.rados.0.smithi190.stdout:97: rollback oid 33 current snap is 4 2023-11-01T04:16:20.023 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 33 to 3 2023-11-01T04:16:20.023 INFO:tasks.rados.rados.0.smithi190.stdout:90: finishing write tid 4 to smithi19039307-17 2023-11-01T04:16:20.024 INFO:tasks.rados.rados.0.smithi190.stdout:90: oid 17 updating version 0 to 6 2023-11-01T04:16:20.024 INFO:tasks.rados.rados.0.smithi190.stdout:90: oid 17 updating version 6 to 8 2023-11-01T04:16:20.024 INFO:tasks.rados.rados.0.smithi190.stdout:90: oid 17 version 8 is already newer than 7 2023-11-01T04:16:20.024 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 17 v 8 (ObjNum 55 snap 3 seq_num 55) dirty exists 2023-11-01T04:16:20.025 INFO:tasks.rados.rados.0.smithi190.stdout:90: left oid 17 (ObjNum 55 snap 3 seq_num 55) 2023-11-01T04:16:20.025 INFO:tasks.rados.rados.0.smithi190.stdout:90: done (4 left) 2023-11-01T04:16:20.025 INFO:tasks.rados.rados.0.smithi190.stdout:98: snap_remove snap 0 2023-11-01T04:16:20.062 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:16:20.059+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:16:20.089 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:20.089 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:20.103 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-01T04:16:20.163 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:20.162+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:20.227 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:20.227 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:20.241 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-01T04:16:20.288 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:20.288+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:20.359 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:20.359 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:20.364 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:20.364+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:20.373 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-01T04:16:20.435 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-01T04:16:20.465 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:20.464+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:20.535 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:20.535 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:20.550 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-01T04:16:20.567 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:20.563+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:20.580 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:20.581 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-11-01T04:16:20.601 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:20.601 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-01T04:16:20.619 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:20.619 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-01T04:16:20.640 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:20.640 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-01T04:16:20.656 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:20.657 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-01T04:16:20.667 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:20.665+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:20.677 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:20.677 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-01T04:16:20.697 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:20.697 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-01T04:16:20.715 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:20.715 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:20.719 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:20.719 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-01T04:16:20.730 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-01T04:16:20.737 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:20.737 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-01T04:16:20.759 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:20.759 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-01T04:16:20.768 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:20.765+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:20.777 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:20.777 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-01T04:16:20.799 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:20.799 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-01T04:16:20.818 INFO:teuthology.orchestra.run.smithi153.stdout:osd.12: {} 2023-11-01T04:16:20.818 INFO:teuthology.orchestra.run.smithi153.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-01T04:16:20.839 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:20.840 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-01T04:16:20.847 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:20.847 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:20.859 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:20.859 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-01T04:16:20.862 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-01T04:16:20.870 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:20.867+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:20.879 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:20.879 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-01T04:16:20.971 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:20.965+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:20.977 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.1 to 1 (10000) 2023-11-01T04:16:20.983 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:20.983 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:16:20.983 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2023-11-01T04:16:20.984 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2023-11-01T04:16:20.984 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:20.985 INFO:tasks.rados.rados.0.smithi190.stdout:98: done (4 left) 2023-11-01T04:16:20.985 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 35 current snap is 4 2023-11-01T04:16:20.986 INFO:tasks.rados.rados.0.smithi190.stdout:99: seq_num 58 ranges {802816=606208,1409024=376832} 2023-11-01T04:16:20.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-01T04:16:20.999 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-11-01T04:16:21.008 INFO:tasks.rados.rados.0.smithi190.stdout:99: writing smithi19039307-35 from 802816 to 1409024 tid 1 2023-11-01T04:16:21.015 INFO:tasks.rados.rados.0.smithi190.stdout:99: writing smithi19039307-35 from 1409024 to 1785856 tid 2 2023-11-01T04:16:21.015 INFO:tasks.rados.rados.0.smithi190.stdout:100: read oid 14 snap -1 2023-11-01T04:16:21.015 INFO:tasks.rados.rados.0.smithi190.stdout:100: expect (ObjNum 38 snap 0 seq_num 38) 2023-11-01T04:16:21.015 INFO:tasks.rados.rados.0.smithi190.stdout:101: snap_create 2023-11-01T04:16:21.016 INFO:tasks.rados.rados.0.smithi190.stdout:94: finishing write tid 1 to smithi19039307-43 2023-11-01T04:16:21.016 INFO:tasks.rados.rados.0.smithi190.stdout:95: finishing write tid 1 to smithi19039307-19 2023-11-01T04:16:21.016 INFO:tasks.rados.rados.0.smithi190.stdout:95: finishing write tid 2 to smithi19039307-19 2023-11-01T04:16:21.016 INFO:tasks.rados.rados.0.smithi190.stdout:95: finishing write tid 3 to smithi19039307-19 2023-11-01T04:16:21.017 INFO:tasks.rados.rados.0.smithi190.stdout:94: finishing write tid 2 to smithi19039307-43 2023-11-01T04:16:21.017 INFO:tasks.rados.rados.0.smithi190.stdout:96: expect (ObjNum 29 snap 0 seq_num 29) 2023-11-01T04:16:21.053 INFO:tasks.rados.rados.0.smithi190.stdout:95: finishing write tid 4 to smithi19039307-19 2023-11-01T04:16:21.054 INFO:tasks.rados.rados.0.smithi190.stdout:95: oid 19 updating version 0 to 9 2023-11-01T04:16:21.054 INFO:tasks.rados.rados.0.smithi190.stdout:95: oid 19 version 9 is already newer than 7 2023-11-01T04:16:21.054 INFO:tasks.rados.rados.0.smithi190.stdout:95: oid 19 version 9 is already newer than 8 2023-11-01T04:16:21.054 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 19 v 9 (ObjNum 57 snap 4 seq_num 57) dirty exists 2023-11-01T04:16:21.055 INFO:tasks.rados.rados.0.smithi190.stdout:95: left oid 19 (ObjNum 57 snap 4 seq_num 57) 2023-11-01T04:16:21.055 INFO:tasks.rados.rados.0.smithi190.stdout:94: finishing write tid 3 to smithi19039307-43 2023-11-01T04:16:21.055 INFO:tasks.rados.rados.0.smithi190.stdout:97: finishing rollback tid 0 to smithi19039307-33 2023-11-01T04:16:21.055 INFO:tasks.rados.rados.0.smithi190.stdout:97: finishing rollback tid 1 to smithi19039307-33 2023-11-01T04:16:21.056 INFO:tasks.rados.rados.0.smithi190.stdout:94: finishing write tid 4 to smithi19039307-43 2023-11-01T04:16:21.056 INFO:tasks.rados.rados.0.smithi190.stdout:94: oid 43 updating version 0 to 12 2023-11-01T04:16:21.056 INFO:tasks.rados.rados.0.smithi190.stdout:94: oid 43 updating version 12 to 13 2023-11-01T04:16:21.056 INFO:tasks.rados.rados.0.smithi190.stdout:94: oid 43 updating version 13 to 14 2023-11-01T04:16:21.056 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 43 v 14 (ObjNum 56 snap 4 seq_num 56) dirty exists 2023-11-01T04:16:21.057 INFO:tasks.rados.rados.0.smithi190.stdout:94: left oid 43 (ObjNum 56 snap 4 seq_num 56) 2023-11-01T04:16:21.057 INFO:tasks.rados.rados.0.smithi190.stdout:97: finishing rollback tid 2 to smithi19039307-33 2023-11-01T04:16:21.057 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 33 v 18 (ObjNum 32 snap 0 seq_num 32) dirty exists 2023-11-01T04:16:21.091 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:21.089+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:21.128 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:21.128 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2023-11-01T04:16:21.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2023-11-01T04:16:21.129 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2023-11-01T04:16:21.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.129 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(osd.13.22:1 3.6s0 3:6b2f2983:::smithi19039307-44:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:21.129 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.940531+0000", 2023-11-01T04:16:21.130 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.1837163559999997, 2023-11-01T04:16:21.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30767406400000002, 2023-11-01T04:16:21.130 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.130 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:21.131 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940531+0000", 2023-11-01T04:16:21.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940531+0000", 2023-11-01T04:16:21.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.133 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.133 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.134 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940532+0000", 2023-11-01T04:16:21.134 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.86e-06 2023-11-01T04:16:21.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940541+0000", 2023-11-01T04:16:21.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7260000000000008e-06 2023-11-01T04:16:21.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940560+0000", 2023-11-01T04:16:21.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8647e-05 2023-11-01T04:16:21.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940571+0000", 2023-11-01T04:16:21.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1688999999999999e-05 2023-11-01T04:16:21.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940733+0000", 2023-11-01T04:16:21.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016122700000000001 2023-11-01T04:16:21.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2023-11-01T04:16:21.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940754+0000", 2023-11-01T04:16:21.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.16e-05 2023-11-01T04:16:21.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.943338+0000", 2023-11-01T04:16:21.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025839080000000002 2023-11-01T04:16:21.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:21.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.943489+0000", 2023-11-01T04:16:21.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151268 2023-11-01T04:16:21.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.248205+0000", 2023-11-01T04:16:21.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30471513900000002 2023-11-01T04:16:21.144 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.144 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.144 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.145 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.6s0 37/31 ECSubRead(tid=24, to_read={3:6b2f2983:::smithi19039307-44:head=0,184320,0}, subchunks={3:6b2f2983:::smithi19039307-44:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:21.145 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.944208+0000", 2023-11-01T04:16:21.146 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.1800392210000004, 2023-11-01T04:16:21.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102755307, 2023-11-01T04:16:21.146 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.146 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.147 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.944208+0000", 2023-11-01T04:16:21.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.944208+0000", 2023-11-01T04:16:21.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.944208+0000", 2023-11-01T04:16:21.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.944208+0000", 2023-11-01T04:16:21.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.152 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.944250+0000", 2023-11-01T04:16:21.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2279e-05 2023-11-01T04:16:21.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.944282+0000", 2023-11-01T04:16:21.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1921999999999997e-05 2023-11-01T04:16:21.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.044571+0000", 2023-11-01T04:16:21.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100288898 2023-11-01T04:16:21.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.046963+0000", 2023-11-01T04:16:21.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023922079999999998 2023-11-01T04:16:21.157 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.157 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.157 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.158 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.6s0 36/31 ECSubReadReply(tid=24, attrs_read=0))", 2023-11-01T04:16:21.158 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.945649+0000", 2023-11-01T04:16:21.159 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.1785980629999999, 2023-11-01T04:16:21.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201734625, 2023-11-01T04:16:21.159 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.159 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.160 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.945649+0000", 2023-11-01T04:16:21.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.945649+0000", 2023-11-01T04:16:21.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.945650+0000", 2023-11-01T04:16:21.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6360000000000001e-06 2023-11-01T04:16:21.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.946159+0000", 2023-11-01T04:16:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00050888500000000004 2023-11-01T04:16:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.946183+0000", 2023-11-01T04:16:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3583000000000001e-05 2023-11-01T04:16:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.946194+0000", 2023-11-01T04:16:21.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1399e-05 2023-11-01T04:16:21.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.147265+0000", 2023-11-01T04:16:21.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20107088000000001 2023-11-01T04:16:21.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.147383+0000", 2023-11-01T04:16:21.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000118242 2023-11-01T04:16:21.169 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.169 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.169 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.170 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:199 3.7s0 3:ec8ceea5:::smithi19039307-25:2 [read 0~409600,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:21.170 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.964921+0000", 2023-11-01T04:16:21.170 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.1593258950000003, 2023-11-01T04:16:21.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.2062832, 2023-11-01T04:16:21.171 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.171 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:21.171 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:21.171 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:21.171 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:21.172 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 199 2023-11-01T04:16:21.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.172 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964921+0000", 2023-11-01T04:16:21.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964921+0000", 2023-11-01T04:16:21.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964924+0000", 2023-11-01T04:16:21.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.593e-06 2023-11-01T04:16:21.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964938+0000", 2023-11-01T04:16:21.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4287000000000001e-05 2023-11-01T04:16:21.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964970+0000", 2023-11-01T04:16:21.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2091999999999997e-05 2023-11-01T04:16:21.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964987+0000", 2023-11-01T04:16:21.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6855999999999999e-05 2023-11-01T04:16:21.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.065231+0000", 2023-11-01T04:16:21.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10024369700000001 2023-11-01T04:16:21.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:21.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.065522+0000", 2023-11-01T04:16:21.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00029172999999999997 2023-11-01T04:16:21.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.171204+0000", 2023-11-01T04:16:21.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105681945 2023-11-01T04:16:21.182 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.183 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.183 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.183 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:200 3.7s0 3:ec8ceea5:::smithi19039307-25:2 [read 0~409600,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:21.184 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.965027+0000", 2023-11-01T04:16:21.184 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.1592202949999999, 2023-11-01T04:16:21.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.207316629, 2023-11-01T04:16:21.184 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.185 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:21.185 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:21.185 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:21.185 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:21.185 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 200 2023-11-01T04:16:21.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.186 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965027+0000", 2023-11-01T04:16:21.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965027+0000", 2023-11-01T04:16:21.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965028+0000", 2023-11-01T04:16:21.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4869999999999999e-06 2023-11-01T04:16:21.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965036+0000", 2023-11-01T04:16:21.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8720000000000002e-06 2023-11-01T04:16:21.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965061+0000", 2023-11-01T04:16:21.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4598999999999999e-05 2023-11-01T04:16:21.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965077+0000", 2023-11-01T04:16:21.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6053999999999999e-05 2023-11-01T04:16:21.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.066413+0000", 2023-11-01T04:16:21.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10133632099999999 2023-11-01T04:16:21.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:21.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.066578+0000", 2023-11-01T04:16:21.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016469100000000001 2023-11-01T04:16:21.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.172343+0000", 2023-11-01T04:16:21.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105765605 2023-11-01T04:16:21.195 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.196 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.196 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.197 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:201 3.7s0 3:ec8ceea5:::smithi19039307-25:2 [read 0~409600,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:21.197 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.965129+0000", 2023-11-01T04:16:21.197 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.1591180789999997, 2023-11-01T04:16:21.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30806539799999999, 2023-11-01T04:16:21.198 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.198 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:21.198 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:21.198 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:21.198 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:21.199 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 201 2023-11-01T04:16:21.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.199 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965129+0000", 2023-11-01T04:16:21.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.201 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.201 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965129+0000", 2023-11-01T04:16:21.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965130+0000", 2023-11-01T04:16:21.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.387e-06 2023-11-01T04:16:21.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965138+0000", 2023-11-01T04:16:21.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8189999999999992e-06 2023-11-01T04:16:21.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965177+0000", 2023-11-01T04:16:21.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8946000000000003e-05 2023-11-01T04:16:21.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965193+0000", 2023-11-01T04:16:21.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5739e-05 2023-11-01T04:16:21.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.067121+0000", 2023-11-01T04:16:21.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101928121 2023-11-01T04:16:21.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:21.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.067291+0000", 2023-11-01T04:16:21.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016974000000000001 2023-11-01T04:16:21.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.273194+0000", 2023-11-01T04:16:21.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.205903646 2023-11-01T04:16:21.209 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.209 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.210 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.210 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.6s0 37/31 ECSubReadReply(tid=24, attrs_read=0))", 2023-11-01T04:16:21.210 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.045775+0000", 2023-11-01T04:16:21.211 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.0784717779999999, 2023-11-01T04:16:21.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201805491, 2023-11-01T04:16:21.211 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.211 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.212 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.045775+0000", 2023-11-01T04:16:21.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.213 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.045775+0000", 2023-11-01T04:16:21.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.045777+0000", 2023-11-01T04:16:21.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6300000000000001e-06 2023-11-01T04:16:21.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.046433+0000", 2023-11-01T04:16:21.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00065657599999999997 2023-11-01T04:16:21.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.046462+0000", 2023-11-01T04:16:21.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8639000000000001e-05 2023-11-01T04:16:21.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.217 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.046481+0000", 2023-11-01T04:16:21.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9035999999999999e-05 2023-11-01T04:16:21.218 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.218 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.247513+0000", 2023-11-01T04:16:21.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20103245 2023-11-01T04:16:21.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.219 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.247581+0000", 2023-11-01T04:16:21.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7160000000000001e-05 2023-11-01T04:16:21.220 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.220 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.220 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.221 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.6s0 37/31 ECSubReadReply(tid=24, attrs_read=0))", 2023-11-01T04:16:21.221 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.046942+0000", 2023-11-01T04:16:21.221 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.0773049229999998, 2023-11-01T04:16:21.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201325435, 2023-11-01T04:16:21.222 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.222 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.222 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.222 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.046942+0000", 2023-11-01T04:16:21.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.046942+0000", 2023-11-01T04:16:21.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.046942+0000", 2023-11-01T04:16:21.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.046942+0000", 2023-11-01T04:16:21.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.226 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.226 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.226 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.047055+0000", 2023-11-01T04:16:21.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011343199999999999 2023-11-01T04:16:21.227 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.227 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.047077+0000", 2023-11-01T04:16:21.228 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1832999999999999e-05 2023-11-01T04:16:21.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.228 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.228 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.247646+0000", 2023-11-01T04:16:21.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.200568359 2023-11-01T04:16:21.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.229 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.248267+0000", 2023-11-01T04:16:21.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000621811 2023-11-01T04:16:21.230 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.230 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.230 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.231 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.7s0 37/31 ECSubReadReply(tid=26, attrs_read=0))", 2023-11-01T04:16:21.231 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.068499+0000", 2023-11-01T04:16:21.231 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.05574776, 2023-11-01T04:16:21.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103901916, 2023-11-01T04:16:21.232 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.232 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.232 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.068499+0000", 2023-11-01T04:16:21.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.233 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.068499+0000", 2023-11-01T04:16:21.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.234 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.234 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.234 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.235 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.068500+0000", 2023-11-01T04:16:21.235 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7300000000000005e-07 2023-11-01T04:16:21.235 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.235 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.235 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.068679+0000", 2023-11-01T04:16:21.236 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017855799999999999 2023-11-01T04:16:21.236 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.236 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.236 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.236 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.068695+0000", 2023-11-01T04:16:21.237 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6759e-05 2023-11-01T04:16:21.237 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.237 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.168765+0000", 2023-11-01T04:16:21.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100069242 2023-11-01T04:16:21.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.171601+0000", 2023-11-01T04:16:21.239 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028365719999999999 2023-11-01T04:16:21.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.239 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.239 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.172401+0000", 2023-11-01T04:16:21.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00079991199999999995 2023-11-01T04:16:21.240 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.240 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.240 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.240 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.241 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.7s0 37/31 ECSubReadReply(tid=27, attrs_read=0))", 2023-11-01T04:16:21.241 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.069040+0000", 2023-11-01T04:16:21.241 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.0552069389999996, 2023-11-01T04:16:21.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102319035, 2023-11-01T04:16:21.242 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.242 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.242 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.242 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.242 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.243 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069040+0000", 2023-11-01T04:16:21.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.243 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.243 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.244 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069040+0000", 2023-11-01T04:16:21.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.244 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.244 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.245 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069041+0000", 2023-11-01T04:16:21.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9700000000000005e-07 2023-11-01T04:16:21.245 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.245 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069267+0000", 2023-11-01T04:16:21.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022573499999999999 2023-11-01T04:16:21.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069290+0000", 2023-11-01T04:16:21.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3642000000000002e-05 2023-11-01T04:16:21.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.247 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.248 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069310+0000", 2023-11-01T04:16:21.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9669000000000001e-05 2023-11-01T04:16:21.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.248 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.249 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.171298+0000", 2023-11-01T04:16:21.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101987984 2023-11-01T04:16:21.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.249 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.171359+0000", 2023-11-01T04:16:21.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1107999999999998e-05 2023-11-01T04:16:21.250 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.250 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.250 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.251 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.7s0 37/31 ECSubReadReply(tid=26, attrs_read=0))", 2023-11-01T04:16:21.251 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.069335+0000", 2023-11-01T04:16:21.252 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.0549114499999996, 2023-11-01T04:16:21.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102231919, 2023-11-01T04:16:21.252 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.252 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.252 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.253 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069335+0000", 2023-11-01T04:16:21.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.253 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.254 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.254 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069335+0000", 2023-11-01T04:16:21.254 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.255 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.255 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069335+0000", 2023-11-01T04:16:21.255 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.255 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.255 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.255 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069335+0000", 2023-11-01T04:16:21.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.257 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069367+0000", 2023-11-01T04:16:21.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1479000000000001e-05 2023-11-01T04:16:21.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.257 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.257 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.258 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069380+0000", 2023-11-01T04:16:21.258 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2952999999999999e-05 2023-11-01T04:16:21.258 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.171500+0000", 2023-11-01T04:16:21.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102120482 2023-11-01T04:16:21.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.171567+0000", 2023-11-01T04:16:21.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7005e-05 2023-11-01T04:16:21.260 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.260 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.260 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.261 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.261 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.7s0 37/31 ECSubReadReply(tid=27, attrs_read=0))", 2023-11-01T04:16:21.261 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.069822+0000", 2023-11-01T04:16:21.261 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.0544249839999997, 2023-11-01T04:16:21.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102734904, 2023-11-01T04:16:21.262 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.262 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.262 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069822+0000", 2023-11-01T04:16:21.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.264 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069822+0000", 2023-11-01T04:16:21.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.264 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.264 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.264 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069822+0000", 2023-11-01T04:16:21.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.265 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069822+0000", 2023-11-01T04:16:21.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.267 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069852+0000", 2023-11-01T04:16:21.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0389e-05 2023-11-01T04:16:21.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.267 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.267 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.069866+0000", 2023-11-01T04:16:21.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3458999999999999e-05 2023-11-01T04:16:21.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.268 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.268 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.172501+0000", 2023-11-01T04:16:21.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102635341 2023-11-01T04:16:21.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.172557+0000", 2023-11-01T04:16:21.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5714999999999998e-05 2023-11-01T04:16:21.270 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.270 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.270 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.271 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.7s0 37/31 ECSubReadReply(tid=27, attrs_read=0))", 2023-11-01T04:16:21.271 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.169717+0000", 2023-11-01T04:16:21.271 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.954529655, 2023-11-01T04:16:21.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10351869599999999, 2023-11-01T04:16:21.271 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.272 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.272 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.272 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.272 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.169717+0000", 2023-11-01T04:16:21.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.273 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.273 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.273 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.273 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.169717+0000", 2023-11-01T04:16:21.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.274 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.274 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.274 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.169718+0000", 2023-11-01T04:16:21.275 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0669999999999999e-06 2023-11-01T04:16:21.275 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.275 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.275 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.275 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.169877+0000", 2023-11-01T04:16:21.275 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015835 2023-11-01T04:16:21.276 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.276 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.276 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.276 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.169896+0000", 2023-11-01T04:16:21.276 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9227000000000001e-05 2023-11-01T04:16:21.277 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.277 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.277 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.169909+0000", 2023-11-01T04:16:21.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2799000000000001e-05 2023-11-01T04:16:21.278 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.278 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.278 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.278 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.272543+0000", 2023-11-01T04:16:21.278 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102634214 2023-11-01T04:16:21.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.279 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.279 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.273236+0000", 2023-11-01T04:16:21.279 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000693039 2023-11-01T04:16:21.279 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.280 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.280 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.280 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.280 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.280 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:214 3.7s0 3:e43a5132:::smithi19039307-40:head [append 0~0] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2023-11-01T04:16:21.281 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.953956+0000", 2023-11-01T04:16:21.281 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.170291126, 2023-11-01T04:16:21.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.310752374, 2023-11-01T04:16:21.281 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.281 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:21.282 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:21.282 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:21.282 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:21.282 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 214 2023-11-01T04:16:21.282 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.282 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.283 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.283 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.283 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.953956+0000", 2023-11-01T04:16:21.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.284 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.284 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.284 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.953956+0000", 2023-11-01T04:16:21.284 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.284 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.285 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.285 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.285 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.953959+0000", 2023-11-01T04:16:21.285 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2890000000000002e-06 2023-11-01T04:16:21.285 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.286 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.286 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.953978+0000", 2023-11-01T04:16:21.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8519999999999999e-05 2023-11-01T04:16:21.286 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954023+0000", 2023-11-01T04:16:21.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5640999999999999e-05 2023-11-01T04:16:21.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.288 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.288 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954048+0000", 2023-11-01T04:16:21.288 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4569000000000001e-05 2023-11-01T04:16:21.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.289 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.289 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954206+0000", 2023-11-01T04:16:21.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015823200000000001 2023-11-01T04:16:21.289 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.289 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2023-11-01T04:16:21.290 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954219+0000", 2023-11-01T04:16:21.290 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3052e-05 2023-11-01T04:16:21.290 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.291 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.055038+0000", 2023-11-01T04:16:21.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10081902299999999 2023-11-01T04:16:21.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:21.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.055200+0000", 2023-11-01T04:16:21.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000162304 2023-11-01T04:16:21.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:21.293 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.056576+0000", 2023-11-01T04:16:21.293 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013755169999999999 2023-11-01T04:16:21.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.294 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:21.294 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063225+0000", 2023-11-01T04:16:21.294 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066492729999999998 2023-11-01T04:16:21.294 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.295 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:21.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.264608+0000", 2023-11-01T04:16:21.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201382965 2023-11-01T04:16:21.295 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.264708+0000", 2023-11-01T04:16:21.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9988999999999994e-05 2023-11-01T04:16:21.296 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.296 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.297 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.297 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.297 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:215 3.7s0 3:e43a5132:::smithi19039307-40:head [rollback 3] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2023-11-01T04:16:21.297 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.954091+0000", 2023-11-01T04:16:21.297 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.1701557509999998, 2023-11-01T04:16:21.298 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.31100268599999997, 2023-11-01T04:16:21.298 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.298 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:21.298 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:21.299 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:21.299 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:21.299 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215 2023-11-01T04:16:21.299 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.299 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.300 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.300 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.300 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954091+0000", 2023-11-01T04:16:21.300 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.300 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.301 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.301 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.301 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954091+0000", 2023-11-01T04:16:21.301 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.301 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.302 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.302 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.302 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954092+0000", 2023-11-01T04:16:21.302 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.33e-07 2023-11-01T04:16:21.303 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.303 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.303 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.303 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954097+0000", 2023-11-01T04:16:21.303 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3770000000000004e-06 2023-11-01T04:16:21.303 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.304 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.304 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.304 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954111+0000", 2023-11-01T04:16:21.304 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3145999999999999e-05 2023-11-01T04:16:21.304 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.305 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.305 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.305 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954123+0000", 2023-11-01T04:16:21.305 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2296000000000001e-05 2023-11-01T04:16:21.305 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.306 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.306 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.306 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954349+0000", 2023-11-01T04:16:21.306 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022646499999999999 2023-11-01T04:16:21.306 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.307 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.307 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:21.307 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954362+0000", 2023-11-01T04:16:21.307 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.312e-05 2023-11-01T04:16:21.307 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.308 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.308 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.308 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.057871+0000", 2023-11-01T04:16:21.308 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10350849299999999 2023-11-01T04:16:21.308 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.309 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.309 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:21.309 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.058109+0000", 2023-11-01T04:16:21.309 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000237631 2023-11-01T04:16:21.309 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:21.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.059576+0000", 2023-11-01T04:16:21.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001467154 2023-11-01T04:16:21.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:21.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063386+0000", 2023-11-01T04:16:21.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038099010000000001 2023-11-01T04:16:21.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.312 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.312 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:21.312 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.265007+0000", 2023-11-01T04:16:21.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20162109 2023-11-01T04:16:21.312 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.313 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.313 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.313 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.265094+0000", 2023-11-01T04:16:21.313 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.708e-05 2023-11-01T04:16:21.313 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.314 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.314 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.314 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.314 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.314 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 37/31 ECSubWriteReply(tid=28, last_complete=38'10, committed=1, applied=1))", 2023-11-01T04:16:21.315 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:18.059672+0000", 2023-11-01T04:16:21.315 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.0645752320000001, 2023-11-01T04:16:21.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103911501, 2023-11-01T04:16:21.315 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.315 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.316 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.316 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.316 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.059672+0000", 2023-11-01T04:16:21.317 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.317 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.317 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.317 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.318 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.059672+0000", 2023-11-01T04:16:21.318 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.318 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.059673+0000", 2023-11-01T04:16:21.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.64e-06 2023-11-01T04:16:21.319 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.319 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.319 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.320 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.059684+0000", 2023-11-01T04:16:21.320 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1158999999999999e-05 2023-11-01T04:16:21.320 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.320 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.320 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.321 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.059712+0000", 2023-11-01T04:16:21.321 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7679e-05 2023-11-01T04:16:21.321 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.321 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.321 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.322 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.059733+0000", 2023-11-01T04:16:21.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0973000000000001e-05 2023-11-01T04:16:21.322 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.322 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.322 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.323 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.163552+0000", 2023-11-01T04:16:21.323 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103819235 2023-11-01T04:16:21.323 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.323 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.323 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.324 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.163583+0000", 2023-11-01T04:16:21.324 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0815000000000002e-05 2023-11-01T04:16:21.324 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.324 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.324 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.325 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.325 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 38/31 ECSubWriteReply(tid=29, last_complete=38'11, committed=1, applied=1))", 2023-11-01T04:16:21.325 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:18.061050+0000", 2023-11-01T04:16:21.325 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.0631968820000002, 2023-11-01T04:16:21.325 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10264403699999999, 2023-11-01T04:16:21.326 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.326 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.326 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.326 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.326 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.061050+0000", 2023-11-01T04:16:21.327 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.327 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.327 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.327 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.061050+0000", 2023-11-01T04:16:21.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.061058+0000", 2023-11-01T04:16:21.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7670000000000003e-06 2023-11-01T04:16:21.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.061066+0000", 2023-11-01T04:16:21.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0199999999999994e-06 2023-11-01T04:16:21.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.330 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.330 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.061091+0000", 2023-11-01T04:16:21.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5255000000000001e-05 2023-11-01T04:16:21.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.061105+0000", 2023-11-01T04:16:21.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4243000000000001e-05 2023-11-01T04:16:21.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.163667+0000", 2023-11-01T04:16:21.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10256127299999999 2023-11-01T04:16:21.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.163694+0000", 2023-11-01T04:16:21.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7478999999999999e-05 2023-11-01T04:16:21.333 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.334 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.334 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.334 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 38/31 ECSubWriteReply(tid=29, last_complete=38'11, committed=1, applied=1))", 2023-11-01T04:16:21.335 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:18.063933+0000", 2023-11-01T04:16:21.335 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.0603135579999998, 2023-11-01T04:16:21.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20052217899999999, 2023-11-01T04:16:21.335 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.335 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.336 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063933+0000", 2023-11-01T04:16:21.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063933+0000", 2023-11-01T04:16:21.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063935+0000", 2023-11-01T04:16:21.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4160000000000001e-06 2023-11-01T04:16:21.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.339 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.339 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063946+0000", 2023-11-01T04:16:21.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0740000000000001e-05 2023-11-01T04:16:21.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063973+0000", 2023-11-01T04:16:21.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7645999999999999e-05 2023-11-01T04:16:21.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.164073+0000", 2023-11-01T04:16:21.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10009936799999999 2023-11-01T04:16:21.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.264433+0000", 2023-11-01T04:16:21.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10036046 2023-11-01T04:16:21.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.264456+0000", 2023-11-01T04:16:21.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2549e-05 2023-11-01T04:16:21.343 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.344 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.344 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.345 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 38/31 ECSubWriteReply(tid=30, last_complete=38'12, committed=1, applied=1))", 2023-11-01T04:16:21.345 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:18.063950+0000", 2023-11-01T04:16:21.345 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.0602972049999999, 2023-11-01T04:16:21.345 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20030165, 2023-11-01T04:16:21.345 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.345 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.346 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.346 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.346 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.346 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063950+0000", 2023-11-01T04:16:21.346 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.347 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.347 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.347 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.347 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063950+0000", 2023-11-01T04:16:21.347 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.347 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.348 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.348 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.348 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063951+0000", 2023-11-01T04:16:21.348 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4169999999999999e-06 2023-11-01T04:16:21.348 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.349 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.349 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.349 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063960+0000", 2023-11-01T04:16:21.349 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4319999999999999e-06 2023-11-01T04:16:21.349 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063981+0000", 2023-11-01T04:16:21.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1946e-05 2023-11-01T04:16:21.350 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.351 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.351 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063998+0000", 2023-11-01T04:16:21.351 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6540000000000001e-05 2023-11-01T04:16:21.351 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.352 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.352 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.352 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.264221+0000", 2023-11-01T04:16:21.352 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20022311600000001 2023-11-01T04:16:21.352 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.352 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.353 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.353 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.264251+0000", 2023-11-01T04:16:21.353 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0199e-05 2023-11-01T04:16:21.353 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.353 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.354 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.354 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 38/31 ECSubWriteReply(tid=30, last_complete=38'12, committed=1, applied=1))", 2023-11-01T04:16:21.354 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:18.065012+0000", 2023-11-01T04:16:21.354 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.0592350160000001, 2023-11-01T04:16:21.355 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.199346354, 2023-11-01T04:16:21.355 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:21.355 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.355 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:21.355 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.356 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:21.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.065012+0000", 2023-11-01T04:16:21.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.357 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:21.357 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.065012+0000", 2023-11-01T04:16:21.357 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:21.357 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.357 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.357 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:21.358 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.065013+0000", 2023-11-01T04:16:21.358 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1599999999999999e-06 2023-11-01T04:16:21.358 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.358 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.358 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:21.359 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.065020+0000", 2023-11-01T04:16:21.359 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5819999999999998e-06 2023-11-01T04:16:21.359 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.359 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.359 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:21.360 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.065039+0000", 2023-11-01T04:16:21.360 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9786999999999999e-05 2023-11-01T04:16:21.360 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.360 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.360 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.361 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.065050+0000", 2023-11-01T04:16:21.361 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0546e-05 2023-11-01T04:16:21.361 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.361 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.361 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:21.362 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.264331+0000", 2023-11-01T04:16:21.362 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19928098999999999 2023-11-01T04:16:21.362 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:21.362 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:21.362 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:21.362 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.264358+0000", 2023-11-01T04:16:21.363 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7288999999999999e-05 2023-11-01T04:16:21.363 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.363 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.363 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.363 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:21.364 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:21.364 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:21.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-11-01T04:16:21.366 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:21.366+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:21.423 INFO:tasks.rados.rados.0.smithi190.stdout:99: finishing write tid 1 to smithi19039307-35 2023-11-01T04:16:21.423 INFO:tasks.rados.rados.0.smithi190.stdout:99: finishing write tid 2 to smithi19039307-35 2023-11-01T04:16:21.429 INFO:tasks.rados.rados.0.smithi190.stdout:100: expect (ObjNum 38 snap 0 seq_num 38) 2023-11-01T04:16:21.466 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:21.464+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:21.500 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:21.500 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2023-11-01T04:16:21.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.501 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 41/31 ECSubWriteReply(tid=16, last_complete=41'10, committed=1, applied=1))", 2023-11-01T04:16:21.501 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.341758+0000", 2023-11-01T04:16:21.501 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.16030328399999999, 2023-11-01T04:16:21.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.160360904, 2023-11-01T04:16:21.501 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.502 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:21.502 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341758+0000", 2023-11-01T04:16:21.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341758+0000", 2023-11-01T04:16:21.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.505 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341760+0000", 2023-11-01T04:16:21.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3629999999999999e-06 2023-11-01T04:16:21.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.506 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341769+0000", 2023-11-01T04:16:21.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4930000000000004e-06 2023-11-01T04:16:21.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341802+0000", 2023-11-01T04:16:21.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2648999999999999e-05 2023-11-01T04:16:21.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441881+0000", 2023-11-01T04:16:21.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100079643 2023-11-01T04:16:21.508 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.508 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.509 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.509 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 41/31 ECSubWriteReply(tid=16, last_complete=41'10, committed=1, applied=1))", 2023-11-01T04:16:21.509 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.341891+0000", 2023-11-01T04:16:21.510 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.16017082099999999, 2023-11-01T04:16:21.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.160311596, 2023-11-01T04:16:21.510 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.510 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:21.510 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341891+0000", 2023-11-01T04:16:21.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.512 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341891+0000", 2023-11-01T04:16:21.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341892+0000", 2023-11-01T04:16:21.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6300000000000001e-06 2023-11-01T04:16:21.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341902+0000", 2023-11-01T04:16:21.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5719999999999992e-06 2023-11-01T04:16:21.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341928+0000", 2023-11-01T04:16:21.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5542e-05 2023-11-01T04:16:21.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341946+0000", 2023-11-01T04:16:21.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8955999999999999e-05 2023-11-01T04:16:21.517 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.517 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.517 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.518 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 41/31 ECSubWriteReply(tid=16, last_complete=41'10, committed=1, applied=1))", 2023-11-01T04:16:21.518 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.342003+0000", 2023-11-01T04:16:21.518 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.16005824499999999, 2023-11-01T04:16:21.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.16026892600000001, 2023-11-01T04:16:21.518 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.519 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:21.519 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.519 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.342003+0000", 2023-11-01T04:16:21.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.342003+0000", 2023-11-01T04:16:21.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.342004+0000", 2023-11-01T04:16:21.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7699999999999992e-07 2023-11-01T04:16:21.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.342011+0000", 2023-11-01T04:16:21.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3629999999999999e-06 2023-11-01T04:16:21.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.342030+0000", 2023-11-01T04:16:21.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9619e-05 2023-11-01T04:16:21.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.342045+0000", 2023-11-01T04:16:21.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4986e-05 2023-11-01T04:16:21.525 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.525 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.525 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.526 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.8s3 41/31 ECSubWrite(tid=22, reqid=osd.5.0:21, at_version=0'0, trim_to=0'0, roll_forward_to=41'12))", 2023-11-01T04:16:21.526 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.441913+0000", 2023-11-01T04:16:21.526 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.060148597999999998, 2023-11-01T04:16:21.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.060433872, 2023-11-01T04:16:21.527 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.527 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "no flag points reached", 2023-11-01T04:16:21.527 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441913+0000", 2023-11-01T04:16:21.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441913+0000", 2023-11-01T04:16:21.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441914+0000", 2023-11-01T04:16:21.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8999999999999995e-07 2023-11-01T04:16:21.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441929+0000", 2023-11-01T04:16:21.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5358000000000001e-05 2023-11-01T04:16:21.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441947+0000", 2023-11-01T04:16:21.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7323e-05 2023-11-01T04:16:21.532 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.532 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.532 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.533 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.533 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2023-11-01T04:16:21.533 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 4 2023-11-01T04:16:21.533 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:21.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-11-01T04:16:21.535 INFO:tasks.rados.rados.0.smithi190.stdout:99: finishing write tid 3 to smithi19039307-35 2023-11-01T04:16:21.567 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:21.566+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:21.654 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2023-11-01T04:16:21.655 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2023-11-01T04:16:21.655 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:21.668 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-11-01T04:16:21.669 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:21.668+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:21.770 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:21.768+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:21.788 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:21.788 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2023-11-01T04:16:21.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2023-11-01T04:16:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2023-11-01T04:16:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:136 3.0s0 3:0e42a959:::smithi19039307-46:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:21.789 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:15.181743+0000", 2023-11-01T04:16:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.607124937, 2023-11-01T04:16:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40000987799999999, 2023-11-01T04:16:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:21.790 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:21.791 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:21.791 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:21.791 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 136 2023-11-01T04:16:21.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.792 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.181743+0000", 2023-11-01T04:16:21.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.181743+0000", 2023-11-01T04:16:21.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.181745+0000", 2023-11-01T04:16:21.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5799999999999999e-06 2023-11-01T04:16:21.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.182798+0000", 2023-11-01T04:16:21.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001052648 2023-11-01T04:16:21.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.182830+0000", 2023-11-01T04:16:21.797 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1786999999999999e-05 2023-11-01T04:16:21.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.797 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.282919+0000", 2023-11-01T04:16:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10008908599999999 2023-11-01T04:16:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283049+0000", 2023-11-01T04:16:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013068500000000001 2023-11-01T04:16:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:21.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283434+0000", 2023-11-01T04:16:21.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038438399999999998 2023-11-01T04:16:21.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.286060+0000", 2023-11-01T04:16:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026267180000000001 2023-11-01T04:16:21.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.802 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:21.802 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.290497+0000", 2023-11-01T04:16:21.802 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044367979999999996 2023-11-01T04:16:21.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.803 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.803 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:21.803 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.581619+0000", 2023-11-01T04:16:21.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29112156700000003 2023-11-01T04:16:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.581752+0000", 2023-11-01T04:16:21.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013362500000000001 2023-11-01T04:16:21.805 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.805 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.805 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.806 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:137 3.0s0 3:0e42a959:::smithi19039307-46:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:21.806 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:15.282962+0000", 2023-11-01T04:16:21.806 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5059054160000001, 2023-11-01T04:16:21.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.32685867899999999, 2023-11-01T04:16:21.806 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.807 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:21.807 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:21.807 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:21.807 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:21.808 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 137 2023-11-01T04:16:21.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.808 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.808 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.282962+0000", 2023-11-01T04:16:21.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.282962+0000", 2023-11-01T04:16:21.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.282964+0000", 2023-11-01T04:16:21.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9269999999999999e-06 2023-11-01T04:16:21.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.282975+0000", 2023-11-01T04:16:21.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0883000000000001e-05 2023-11-01T04:16:21.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.282997+0000", 2023-11-01T04:16:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1897999999999999e-05 2023-11-01T04:16:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283012+0000", 2023-11-01T04:16:21.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5682999999999999e-05 2023-11-01T04:16:21.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:21.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.287942+0000", 2023-11-01T04:16:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049297919999999997 2023-11-01T04:16:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:21.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.288248+0000", 2023-11-01T04:16:21.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030577300000000001 2023-11-01T04:16:21.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:21.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.289532+0000", 2023-11-01T04:16:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012844060000000001 2023-11-01T04:16:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:21.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.290865+0000", 2023-11-01T04:16:21.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013323320000000001 2023-11-01T04:16:21.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:21.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.609653+0000", 2023-11-01T04:16:21.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31878868999999999 2023-11-01T04:16:21.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.609821+0000", 2023-11-01T04:16:21.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000167295 2023-11-01T04:16:21.829 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.829 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.829 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.830 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:138 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:15.283030+0000", 2023-11-01T04:16:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5058374260000003, 2023-11-01T04:16:21.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52972434099999999, 2023-11-01T04:16:21.832 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.832 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:21.833 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:21.833 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:21.833 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 138 2023-11-01T04:16:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.834 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283030+0000", 2023-11-01T04:16:21.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.837 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283030+0000", 2023-11-01T04:16:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283031+0000", 2023-11-01T04:16:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9600000000000008e-07 2023-11-01T04:16:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.841 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.841 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283035+0000", 2023-11-01T04:16:21.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7299999999999999e-06 2023-11-01T04:16:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283046+0000", 2023-11-01T04:16:21.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1249e-05 2023-11-01T04:16:21.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.844 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283055+0000", 2023-11-01T04:16:21.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2660000000000007e-06 2023-11-01T04:16:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.846 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:21.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.290158+0000", 2023-11-01T04:16:21.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0071024260000000002 2023-11-01T04:16:21.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.848 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.848 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for rw locks", 2023-11-01T04:16:21.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.290320+0000", 2023-11-01T04:16:21.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000161886 2023-11-01T04:16:21.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.850 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.850 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:21.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.610629+0000", 2023-11-01T04:16:21.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.32030918600000002 2023-11-01T04:16:21.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.852 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:21.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.610871+0000", 2023-11-01T04:16:21.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000242124 2023-11-01T04:16:21.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:21.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.812754+0000", 2023-11-01T04:16:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20188347800000001 2023-11-01T04:16:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.855 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.856 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.857 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:161 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~770048,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.318323+0000", 2023-11-01T04:16:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.4705446860000002, 2023-11-01T04:16:21.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20523274599999999, 2023-11-01T04:16:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:21.859 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:21.860 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:21.860 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 161 2023-11-01T04:16:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.861 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318323+0000", 2023-11-01T04:16:21.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318323+0000", 2023-11-01T04:16:21.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318327+0000", 2023-11-01T04:16:21.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.726e-06 2023-11-01T04:16:21.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318354+0000", 2023-11-01T04:16:21.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7404999999999999e-05 2023-11-01T04:16:21.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318523+0000", 2023-11-01T04:16:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016939900000000001 2023-11-01T04:16:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318550+0000", 2023-11-01T04:16:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6338000000000001e-05 2023-11-01T04:16:21.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318678+0000", 2023-11-01T04:16:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000127899 2023-11-01T04:16:21.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318814+0000", 2023-11-01T04:16:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013687399999999999 2023-11-01T04:16:21.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.523556+0000", 2023-11-01T04:16:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20474110500000001 2023-11-01T04:16:21.874 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:162 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~770048,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:21.875 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.318603+0000", 2023-11-01T04:16:21.876 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.4702646340000003, 2023-11-01T04:16:21.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.206224885, 2023-11-01T04:16:21.876 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.876 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:21.876 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 162 2023-11-01T04:16:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.877 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318603+0000", 2023-11-01T04:16:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318603+0000", 2023-11-01T04:16:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318605+0000", 2023-11-01T04:16:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.903e-06 2023-11-01T04:16:21.880 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318614+0000", 2023-11-01T04:16:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6360000000000006e-06 2023-11-01T04:16:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318637+0000", 2023-11-01T04:16:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2617999999999999e-05 2023-11-01T04:16:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318657+0000", 2023-11-01T04:16:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9551999999999998e-05 2023-11-01T04:16:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.319458+0000", 2023-11-01T04:16:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00080107500000000005 2023-11-01T04:16:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.319692+0000", 2023-11-01T04:16:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000234675 2023-11-01T04:16:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.524828+0000", 2023-11-01T04:16:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20513542600000001 2023-11-01T04:16:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.886 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:163 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~770048,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.318690+0000", 2023-11-01T04:16:21.887 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.4701778399999998, 2023-11-01T04:16:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.309365155, 2023-11-01T04:16:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:21.888 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 163 2023-11-01T04:16:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318690+0000", 2023-11-01T04:16:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318690+0000", 2023-11-01T04:16:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318691+0000", 2023-11-01T04:16:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.263e-06 2023-11-01T04:16:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318698+0000", 2023-11-01T04:16:21.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7900000000000002e-06 2023-11-01T04:16:21.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.894 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318716+0000", 2023-11-01T04:16:21.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8128e-05 2023-11-01T04:16:21.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.894 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.894 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.895 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.418889+0000", 2023-11-01T04:16:21.895 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100173525 2023-11-01T04:16:21.895 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.895 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:21.896 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.524945+0000", 2023-11-01T04:16:21.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.106055714 2023-11-01T04:16:21.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.896 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.896 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:21.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.525089+0000", 2023-11-01T04:16:21.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014418800000000001 2023-11-01T04:16:21.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:21.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.628055+0000", 2023-11-01T04:16:21.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.102965547 2023-11-01T04:16:21.898 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.898 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.898 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.899 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 36/31 ECSubReadReply(tid=8, attrs_read=0))", 2023-11-01T04:16:21.899 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.321399+0000", 2023-11-01T04:16:21.899 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.4674687820000001, 2023-11-01T04:16:21.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20100272, 2023-11-01T04:16:21.900 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.900 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.900 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.321399+0000", 2023-11-01T04:16:21.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.902 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.902 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.321399+0000", 2023-11-01T04:16:21.902 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.902 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.903 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.903 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.321400+0000", 2023-11-01T04:16:21.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7430000000000001e-06 2023-11-01T04:16:21.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.904 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322320+0000", 2023-11-01T04:16:21.904 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000919454 2023-11-01T04:16:21.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.905 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322347+0000", 2023-11-01T04:16:21.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6758e-05 2023-11-01T04:16:21.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322364+0000", 2023-11-01T04:16:21.906 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6901999999999998e-05 2023-11-01T04:16:21.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:21.907 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.522338+0000", 2023-11-01T04:16:21.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.199974127 2023-11-01T04:16:21.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.907 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:21.908 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.522401+0000", 2023-11-01T04:16:21.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3736000000000001e-05 2023-11-01T04:16:21.908 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.908 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.908 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.909 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 36/31 ECSubReadReply(tid=9, attrs_read=0))", 2023-11-01T04:16:21.909 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.322400+0000", 2023-11-01T04:16:21.910 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.4664678670000004, 2023-11-01T04:16:21.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20014336599999999, 2023-11-01T04:16:21.910 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.910 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.910 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.911 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322400+0000", 2023-11-01T04:16:21.911 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322400+0000", 2023-11-01T04:16:21.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.913 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.913 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.913 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322401+0000", 2023-11-01T04:16:21.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1599999999999999e-06 2023-11-01T04:16:21.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322751+0000", 2023-11-01T04:16:21.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000349897 2023-11-01T04:16:21.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322777+0000", 2023-11-01T04:16:21.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6171999999999999e-05 2023-11-01T04:16:21.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.916 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322791+0000", 2023-11-01T04:16:21.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3939e-05 2023-11-01T04:16:21.917 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:21.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.522487+0000", 2023-11-01T04:16:21.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.199695822 2023-11-01T04:16:21.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:21.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.522543+0000", 2023-11-01T04:16:21.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6375999999999997e-05 2023-11-01T04:16:21.919 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.919 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.919 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.920 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 36/31 ECSubReadReply(tid=8, attrs_read=0))", 2023-11-01T04:16:21.920 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.420100+0000", 2023-11-01T04:16:21.921 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.3687670660000002, 2023-11-01T04:16:21.921 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10349056099999999, 2023-11-01T04:16:21.921 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.921 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.921 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.922 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.420100+0000", 2023-11-01T04:16:21.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.420100+0000", 2023-11-01T04:16:21.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.923 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.924 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.420102+0000", 2023-11-01T04:16:21.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.697e-06 2023-11-01T04:16:21.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.925 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.420994+0000", 2023-11-01T04:16:21.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00089149599999999998 2023-11-01T04:16:21.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.421015+0000", 2023-11-01T04:16:21.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1151000000000001e-05 2023-11-01T04:16:21.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.421026+0000", 2023-11-01T04:16:21.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0923e-05 2023-11-01T04:16:21.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:21.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.522725+0000", 2023-11-01T04:16:21.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101699766 2023-11-01T04:16:21.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:21.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.523591+0000", 2023-11-01T04:16:21.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00086552799999999996 2023-11-01T04:16:21.929 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.930 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.930 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.930 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es5 37/31 ECSubWrite(tid=20, reqid=client.4559.0:185, at_version=37'13, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:16:21.931 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966469+0000", 2023-11-01T04:16:21.931 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.8223987680000002, 2023-11-01T04:16:21.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.104017606, 2023-11-01T04:16:21.931 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.931 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.932 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966469+0000", 2023-11-01T04:16:21.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966469+0000", 2023-11-01T04:16:21.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966469+0000", 2023-11-01T04:16:21.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6599999999999995e-07 2023-11-01T04:16:21.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966593+0000", 2023-11-01T04:16:21.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012348599999999999 2023-11-01T04:16:21.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.936 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966655+0000", 2023-11-01T04:16:21.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1969000000000002e-05 2023-11-01T04:16:21.936 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.066851+0000", 2023-11-01T04:16:21.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100196493 2023-11-01T04:16:21.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:21.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.067024+0000", 2023-11-01T04:16:21.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017216200000000001 2023-11-01T04:16:21.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:21.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.067048+0000", 2023-11-01T04:16:21.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4345000000000002e-05 2023-11-01T04:16:21.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.940 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:21.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.070411+0000", 2023-11-01T04:16:21.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0033630719999999999 2023-11-01T04:16:21.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.941 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:21.942 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.070486+0000", 2023-11-01T04:16:21.942 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5313000000000002e-05 2023-11-01T04:16:21.942 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.942 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.942 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.943 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.4s5 37/31 ECSubWrite(tid=21, reqid=client.4559.0:196, at_version=37'8, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:21.943 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.038904+0000", 2023-11-01T04:16:21.943 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.7499637129999996, 2023-11-01T04:16:21.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10286664500000001, 2023-11-01T04:16:21.944 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.944 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.944 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.038904+0000", 2023-11-01T04:16:21.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.946 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.038904+0000", 2023-11-01T04:16:21.946 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.946 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.946 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.947 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.038904+0000", 2023-11-01T04:16:21.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9999999999999998e-07 2023-11-01T04:16:21.947 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.947 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.948 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.038947+0000", 2023-11-01T04:16:21.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2370000000000003e-05 2023-11-01T04:16:21.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.038975+0000", 2023-11-01T04:16:21.949 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8388e-05 2023-11-01T04:16:21.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.949 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.139117+0000", 2023-11-01T04:16:21.950 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100141946 2023-11-01T04:16:21.950 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:21.951 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.139284+0000", 2023-11-01T04:16:21.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016725900000000001 2023-11-01T04:16:21.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.951 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.951 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:21.952 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.139333+0000", 2023-11-01T04:16:21.952 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9246999999999998e-05 2023-11-01T04:16:21.952 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.952 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.952 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:21.952 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.141668+0000", 2023-11-01T04:16:21.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023342739999999999 2023-11-01T04:16:21.953 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:21.953 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.141770+0000", 2023-11-01T04:16:21.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000102761 2023-11-01T04:16:21.954 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.954 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.954 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.955 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.as4 41/31 ECSubWrite(tid=54, reqid=osd.12.0:54, at_version=41'11, trim_to=0'0, roll_forward_to=40'9))", 2023-11-01T04:16:21.955 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.116685+0000", 2023-11-01T04:16:21.955 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.67218251299999998, 2023-11-01T04:16:21.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.104963443, 2023-11-01T04:16:21.956 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.956 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.956 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.956 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.956 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.116685+0000", 2023-11-01T04:16:21.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.116685+0000", 2023-11-01T04:16:21.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.116685+0000", 2023-11-01T04:16:21.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4999999999999998e-07 2023-11-01T04:16:21.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.116728+0000", 2023-11-01T04:16:21.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2277000000000002e-05 2023-11-01T04:16:21.960 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.960 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.960 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.116767+0000", 2023-11-01T04:16:21.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9150999999999998e-05 2023-11-01T04:16:21.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.216944+0000", 2023-11-01T04:16:21.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10017764699999999 2023-11-01T04:16:21.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:21.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.219954+0000", 2023-11-01T04:16:21.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030099900000000001 2023-11-01T04:16:21.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:21.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.219969+0000", 2023-11-01T04:16:21.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4306e-05 2023-11-01T04:16:21.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:21.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.221598+0000", 2023-11-01T04:16:21.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001629405 2023-11-01T04:16:21.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:21.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.221648+0000", 2023-11-01T04:16:21.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0317e-05 2023-11-01T04:16:21.966 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.966 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.966 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.967 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.9s6 41/31 ECSubWrite(tid=27, reqid=osd.7.0:27, at_version=41'20, trim_to=0'0, roll_forward_to=40'18))", 2023-11-01T04:16:21.967 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.129794+0000", 2023-11-01T04:16:21.967 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.65907347999999999, 2023-11-01T04:16:21.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.140144983, 2023-11-01T04:16:21.968 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.968 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.968 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.129794+0000", 2023-11-01T04:16:21.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.970 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.970 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.129794+0000", 2023-11-01T04:16:21.970 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.970 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.129796+0000", 2023-11-01T04:16:21.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.243e-06 2023-11-01T04:16:21.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.129847+0000", 2023-11-01T04:16:21.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0606999999999998e-05 2023-11-01T04:16:21.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.129877+0000", 2023-11-01T04:16:21.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0205e-05 2023-11-01T04:16:21.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.129902+0000", 2023-11-01T04:16:21.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4505e-05 2023-11-01T04:16:21.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:21.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.170053+0000", 2023-11-01T04:16:21.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.040151216000000003 2023-11-01T04:16:21.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:21.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.170078+0000", 2023-11-01T04:16:21.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4862e-05 2023-11-01T04:16:21.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:21.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.269876+0000", 2023-11-01T04:16:21.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.099797992000000002 2023-11-01T04:16:21.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:21.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.269939+0000", 2023-11-01T04:16:21.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3353000000000001e-05 2023-11-01T04:16:21.978 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.978 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.978 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.979 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.9s6 41/31 ECSubWrite(tid=28, reqid=osd.7.0:28, at_version=41'22, trim_to=0'0, roll_forward_to=40'18))", 2023-11-01T04:16:21.979 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.132070+0000", 2023-11-01T04:16:21.979 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.65679698600000003, 2023-11-01T04:16:21.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.13790775399999999, 2023-11-01T04:16:21.980 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.980 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.980 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.981 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.981 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.132070+0000", 2023-11-01T04:16:21.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.132070+0000", 2023-11-01T04:16:21.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.132072+0000", 2023-11-01T04:16:21.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.73e-06 2023-11-01T04:16:21.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.132179+0000", 2023-11-01T04:16:21.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010670699999999999 2023-11-01T04:16:21.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.985 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.132223+0000", 2023-11-01T04:16:21.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3727000000000003e-05 2023-11-01T04:16:21.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.132246+0000", 2023-11-01T04:16:21.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3546000000000001e-05 2023-11-01T04:16:21.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:21.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.170907+0000", 2023-11-01T04:16:21.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.038661170000000002 2023-11-01T04:16:21.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:21.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.170924+0000", 2023-11-01T04:16:21.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6409000000000002e-05 2023-11-01T04:16:21.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.989 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:21.989 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.269946+0000", 2023-11-01T04:16:21.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.099022473 2023-11-01T04:16:21.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.990 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:21.990 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.269978+0000", 2023-11-01T04:16:21.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1992000000000001e-05 2023-11-01T04:16:21.990 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.990 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:21.991 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:21.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.991 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s1 41/31 ECSubWrite(tid=218, reqid=osd.13.0:218, at_version=41'23, trim_to=0'0, roll_forward_to=37'21))", 2023-11-01T04:16:21.991 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.236289+0000", 2023-11-01T04:16:21.992 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.55257816500000001, 2023-11-01T04:16:21.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.107363111, 2023-11-01T04:16:21.992 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:21.992 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:21.992 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:21.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:21.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.236289+0000", 2023-11-01T04:16:21.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:21.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.236289+0000", 2023-11-01T04:16:21.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:21.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:21.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.236290+0000", 2023-11-01T04:16:21.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7399999999999999e-07 2023-11-01T04:16:21.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:21.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.236324+0000", 2023-11-01T04:16:21.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4490999999999999e-05 2023-11-01T04:16:21.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:21.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.236352+0000", 2023-11-01T04:16:21.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7708e-05 2023-11-01T04:16:21.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:21.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.236368+0000", 2023-11-01T04:16:21.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5815999999999999e-05 2023-11-01T04:16:21.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:21.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:21.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:21.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.340003+0000", 2023-11-01T04:16:21.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.103635672 2023-11-01T04:16:21.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:22.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.340027+0000", 2023-11-01T04:16:22.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3655e-05 2023-11-01T04:16:22.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:22.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.343589+0000", 2023-11-01T04:16:22.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0035620019999999999 2023-11-01T04:16:22.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.343652+0000", 2023-11-01T04:16:22.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3392999999999994e-05 2023-11-01T04:16:22.003 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.003 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.003 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.004 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s1 41/31 ECSubWrite(tid=219, reqid=osd.13.0:219, at_version=41'25, trim_to=0'0, roll_forward_to=37'21))", 2023-11-01T04:16:22.004 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.239172+0000", 2023-11-01T04:16:22.004 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.54969580799999995, 2023-11-01T04:16:22.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.105465711, 2023-11-01T04:16:22.005 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.005 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.005 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.239172+0000", 2023-11-01T04:16:22.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.239172+0000", 2023-11-01T04:16:22.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.239173+0000", 2023-11-01T04:16:22.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6300000000000001e-06 2023-11-01T04:16:22.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.239240+0000", 2023-11-01T04:16:22.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6963000000000002e-05 2023-11-01T04:16:22.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.239272+0000", 2023-11-01T04:16:22.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2138000000000003e-05 2023-11-01T04:16:22.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.239334+0000", 2023-11-01T04:16:22.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1859000000000005e-05 2023-11-01T04:16:22.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341072+0000", 2023-11-01T04:16:22.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101737654 2023-11-01T04:16:22.012 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:22.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341095+0000", 2023-11-01T04:16:22.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3422e-05 2023-11-01T04:16:22.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:22.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.344516+0000", 2023-11-01T04:16:22.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0034210120000000002 2023-11-01T04:16:22.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.344637+0000", 2023-11-01T04:16:22.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000121033 2023-11-01T04:16:22.015 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.015 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.015 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.016 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 41/31 ECSubWriteReply(tid=16, last_complete=41'10, committed=1, applied=1))", 2023-11-01T04:16:22.016 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.341758+0000", 2023-11-01T04:16:22.016 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.44710916000000001, 2023-11-01T04:16:22.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20006660200000001, 2023-11-01T04:16:22.017 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.017 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.017 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341758+0000", 2023-11-01T04:16:22.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341758+0000", 2023-11-01T04:16:22.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341760+0000", 2023-11-01T04:16:22.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3629999999999999e-06 2023-11-01T04:16:22.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341769+0000", 2023-11-01T04:16:22.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4930000000000004e-06 2023-11-01T04:16:22.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341802+0000", 2023-11-01T04:16:22.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2648999999999999e-05 2023-11-01T04:16:22.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441881+0000", 2023-11-01T04:16:22.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100079643 2023-11-01T04:16:22.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.541800+0000", 2023-11-01T04:16:22.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.099919015 2023-11-01T04:16:22.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.541825+0000", 2023-11-01T04:16:22.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4439000000000001e-05 2023-11-01T04:16:22.025 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.025 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.025 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.026 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 41/31 ECSubWriteReply(tid=16, last_complete=41'10, committed=1, applied=1))", 2023-11-01T04:16:22.026 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.341891+0000", 2023-11-01T04:16:22.026 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.44697669699999998, 2023-11-01T04:16:22.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.19983851699999999, 2023-11-01T04:16:22.027 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.027 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.027 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341891+0000", 2023-11-01T04:16:22.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341891+0000", 2023-11-01T04:16:22.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341892+0000", 2023-11-01T04:16:22.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6300000000000001e-06 2023-11-01T04:16:22.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341902+0000", 2023-11-01T04:16:22.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5719999999999992e-06 2023-11-01T04:16:22.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341928+0000", 2023-11-01T04:16:22.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5542e-05 2023-11-01T04:16:22.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.341946+0000", 2023-11-01T04:16:22.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8955999999999999e-05 2023-11-01T04:16:22.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.034 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.541672+0000", 2023-11-01T04:16:22.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.19972553400000001 2023-11-01T04:16:22.034 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.034 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.034 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.034 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.541729+0000", 2023-11-01T04:16:22.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7283e-05 2023-11-01T04:16:22.035 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.035 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.035 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.036 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 41/31 ECSubWriteReply(tid=16, last_complete=41'10, committed=1, applied=1))", 2023-11-01T04:16:22.036 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.342003+0000", 2023-11-01T04:16:22.036 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.446864121, 2023-11-01T04:16:22.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.199766002, 2023-11-01T04:16:22.036 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.037 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.037 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.342003+0000", 2023-11-01T04:16:22.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.342003+0000", 2023-11-01T04:16:22.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.039 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.342004+0000", 2023-11-01T04:16:22.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7699999999999992e-07 2023-11-01T04:16:22.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.040 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.040 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.040 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.342011+0000", 2023-11-01T04:16:22.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3629999999999999e-06 2023-11-01T04:16:22.041 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.342030+0000", 2023-11-01T04:16:22.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9619e-05 2023-11-01T04:16:22.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.042 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.042 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.342045+0000", 2023-11-01T04:16:22.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4986e-05 2023-11-01T04:16:22.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.541756+0000", 2023-11-01T04:16:22.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.19971077100000001 2023-11-01T04:16:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.541769+0000", 2023-11-01T04:16:22.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3286e-05 2023-11-01T04:16:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.045 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.8s3 41/31 ECSubWrite(tid=22, reqid=osd.5.0:21, at_version=0'0, trim_to=0'0, roll_forward_to=41'12))", 2023-11-01T04:16:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.441913+0000", 2023-11-01T04:16:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.34695447400000001, 2023-11-01T04:16:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20159738899999999, 2023-11-01T04:16:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.046 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441913+0000", 2023-11-01T04:16:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441913+0000", 2023-11-01T04:16:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441914+0000", 2023-11-01T04:16:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8999999999999995e-07 2023-11-01T04:16:22.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441929+0000", 2023-11-01T04:16:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5358000000000001e-05 2023-11-01T04:16:22.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441947+0000", 2023-11-01T04:16:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7323e-05 2023-11-01T04:16:22.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.542038+0000", 2023-11-01T04:16:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10009142 2023-11-01T04:16:22.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.642225+0000", 2023-11-01T04:16:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100186991 2023-11-01T04:16:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:22.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.642241+0000", 2023-11-01T04:16:22.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6302000000000001e-05 2023-11-01T04:16:22.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:22.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.643431+0000", 2023-11-01T04:16:22.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011899359999999999 2023-11-01T04:16:22.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.643510+0000", 2023-11-01T04:16:22.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9169e-05 2023-11-01T04:16:22.056 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.056 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.056 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.057 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.057 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.057 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:22.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-11-01T04:16:22.060 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.2 to 1 (10000) 2023-11-01T04:16:22.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-01T04:16:22.061 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-11-01T04:16:22.061 INFO:tasks.rados.rados.0.smithi190.stdout:99: finishing write tid 4 to smithi19039307-35 2023-11-01T04:16:22.062 INFO:tasks.rados.rados.0.smithi190.stdout:99: oid 35 updating version 0 to 22 2023-11-01T04:16:22.062 INFO:tasks.rados.rados.0.smithi190.stdout:99: oid 35 updating version 22 to 23 2023-11-01T04:16:22.062 INFO:tasks.rados.rados.0.smithi190.stdout:99: oid 35 version 23 is already newer than 21 2023-11-01T04:16:22.062 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 35 v 23 (ObjNum 58 snap 4 seq_num 58) dirty exists 2023-11-01T04:16:22.063 INFO:tasks.rados.rados.0.smithi190.stdout:99: left oid 35 (ObjNum 58 snap 4 seq_num 58) 2023-11-01T04:16:22.063 INFO:tasks.rados.rados.0.smithi190.stdout:94: done (6 left) 2023-11-01T04:16:22.063 INFO:tasks.rados.rados.0.smithi190.stdout:95: done (5 left) 2023-11-01T04:16:22.063 INFO:tasks.rados.rados.0.smithi190.stdout:96: done (4 left) 2023-11-01T04:16:22.063 INFO:tasks.rados.rados.0.smithi190.stdout:97: done (3 left) 2023-11-01T04:16:22.064 INFO:tasks.rados.rados.0.smithi190.stdout:99: done (2 left) 2023-11-01T04:16:22.064 INFO:tasks.rados.rados.0.smithi190.stdout:100: done (1 left) 2023-11-01T04:16:22.064 INFO:tasks.rados.rados.0.smithi190.stdout:101: done (0 left) 2023-11-01T04:16:22.064 INFO:tasks.rados.rados.0.smithi190.stdout:102: snap_remove snap 1 2023-11-01T04:16:22.064 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:22.055+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:22.159 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:22.157+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:22.234 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:22.234 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:22.235 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2023-11-01T04:16:22.235 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:22.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-01T04:16:22.295 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:22.295+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:22.361 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:22.362+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:22.371 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2023-11-01T04:16:22.372 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2023-11-01T04:16:22.372 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:22.386 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-01T04:16:22.462 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:22.457+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:22.553 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:22.553 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2023-11-01T04:16:22.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2023-11-01T04:16:22.554 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2023-11-01T04:16:22.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.554 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:178 3.9s0 3:94ead3c8:::smithi19039307-33:head [read 0~540672,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:22.554 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.939732+0000", 2023-11-01T04:16:22.555 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.6124797009999998, 2023-11-01T04:16:22.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30648703300000002, 2023-11-01T04:16:22.555 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.555 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:22.555 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:22.556 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:22.556 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:22.556 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 178 2023-11-01T04:16:22.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.557 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939732+0000", 2023-11-01T04:16:22.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939732+0000", 2023-11-01T04:16:22.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939735+0000", 2023-11-01T04:16:22.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6199999999999999e-06 2023-11-01T04:16:22.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939748+0000", 2023-11-01T04:16:22.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2704999999999999e-05 2023-11-01T04:16:22.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939780+0000", 2023-11-01T04:16:22.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2478000000000003e-05 2023-11-01T04:16:22.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939796+0000", 2023-11-01T04:16:22.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5492999999999999e-05 2023-11-01T04:16:22.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939948+0000", 2023-11-01T04:16:22.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015278000000000001 2023-11-01T04:16:22.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:22.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.940137+0000", 2023-11-01T04:16:22.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018902400000000001 2023-11-01T04:16:22.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.246219+0000", 2023-11-01T04:16:22.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30608193299999997 2023-11-01T04:16:22.566 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.566 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.566 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.567 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:179 3.9s0 3:94ead3c8:::smithi19039307-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:22.567 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.939832+0000", 2023-11-01T04:16:22.567 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.6123801379999998, 2023-11-01T04:16:22.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30729425199999999, 2023-11-01T04:16:22.568 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.568 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:22.568 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:22.568 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:22.569 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:22.569 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 179 2023-11-01T04:16:22.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.569 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939832+0000", 2023-11-01T04:16:22.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939832+0000", 2023-11-01T04:16:22.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939833+0000", 2023-11-01T04:16:22.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2240000000000001e-06 2023-11-01T04:16:22.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939843+0000", 2023-11-01T04:16:22.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5659999999999996e-06 2023-11-01T04:16:22.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939864+0000", 2023-11-01T04:16:22.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1098e-05 2023-11-01T04:16:22.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939882+0000", 2023-11-01T04:16:22.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8031999999999999e-05 2023-11-01T04:16:22.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.940794+0000", 2023-11-01T04:16:22.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00091202099999999995 2023-11-01T04:16:22.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:22.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.941019+0000", 2023-11-01T04:16:22.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022529900000000001 2023-11-01T04:16:22.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.247126+0000", 2023-11-01T04:16:22.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30610701200000001 2023-11-01T04:16:22.579 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.579 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.580 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.580 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:180 3.9s0 3:94ead3c8:::smithi19039307-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:22.580 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.939918+0000", 2023-11-01T04:16:22.581 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.6122945269999995, 2023-11-01T04:16:22.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30792408900000001, 2023-11-01T04:16:22.581 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.581 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:22.581 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:22.582 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:22.582 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:22.582 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 180 2023-11-01T04:16:22.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.582 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939918+0000", 2023-11-01T04:16:22.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.584 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939918+0000", 2023-11-01T04:16:22.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939919+0000", 2023-11-01T04:16:22.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.187e-06 2023-11-01T04:16:22.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939927+0000", 2023-11-01T04:16:22.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0059999999999992e-06 2023-11-01T04:16:22.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939951+0000", 2023-11-01T04:16:22.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3759000000000001e-05 2023-11-01T04:16:22.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939969+0000", 2023-11-01T04:16:22.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8402000000000001e-05 2023-11-01T04:16:22.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.943066+0000", 2023-11-01T04:16:22.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003097266 2023-11-01T04:16:22.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:22.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.943281+0000", 2023-11-01T04:16:22.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021503900000000001 2023-11-01T04:16:22.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.247842+0000", 2023-11-01T04:16:22.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30456042999999999 2023-11-01T04:16:22.591 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.591 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.592 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.592 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.9s0 36/31 ECSubReadReply(tid=10, attrs_read=0))", 2023-11-01T04:16:22.592 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.942329+0000", 2023-11-01T04:16:22.593 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.6098830230000001, 2023-11-01T04:16:22.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30392635899999998, 2023-11-01T04:16:22.593 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.593 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.593 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.942329+0000", 2023-11-01T04:16:22.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.942329+0000", 2023-11-01T04:16:22.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.942331+0000", 2023-11-01T04:16:22.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.584e-06 2023-11-01T04:16:22.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.942856+0000", 2023-11-01T04:16:22.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000525856 2023-11-01T04:16:22.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.942886+0000", 2023-11-01T04:16:22.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9471e-05 2023-11-01T04:16:22.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.042979+0000", 2023-11-01T04:16:22.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100092548 2023-11-01T04:16:22.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.600 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.245459+0000", 2023-11-01T04:16:22.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20248086400000001 2023-11-01T04:16:22.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.246255+0000", 2023-11-01T04:16:22.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00079603599999999997 2023-11-01T04:16:22.601 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.602 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.602 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.602 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.9s0 37/31 ECSubReadReply(tid=12, attrs_read=0))", 2023-11-01T04:16:22.603 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.945509+0000", 2023-11-01T04:16:22.603 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.6067026289999999, 2023-11-01T04:16:22.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29986526099999999, 2023-11-01T04:16:22.603 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.603 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.604 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.604 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.945509+0000", 2023-11-01T04:16:22.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.945509+0000", 2023-11-01T04:16:22.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.945509+0000", 2023-11-01T04:16:22.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.945509+0000", 2023-11-01T04:16:22.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.608 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.608 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.945548+0000", 2023-11-01T04:16:22.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8074999999999997e-05 2023-11-01T04:16:22.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.945559+0000", 2023-11-01T04:16:22.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1576000000000001e-05 2023-11-01T04:16:22.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.245305+0000", 2023-11-01T04:16:22.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.299746134 2023-11-01T04:16:22.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.245375+0000", 2023-11-01T04:16:22.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9475999999999997e-05 2023-11-01T04:16:22.612 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.612 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.612 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.613 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:181 3.9s0 3:959ff18f:::smithi19039307-32:head [append 0~524288] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:22.613 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.957492+0000", 2023-11-01T04:16:22.613 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.5947205709999999, 2023-11-01T04:16:22.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.62014424999999995, 2023-11-01T04:16:22.614 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.614 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:22.614 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:22.614 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:22.614 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:22.615 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 181 2023-11-01T04:16:22.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.615 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.957492+0000", 2023-11-01T04:16:22.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.957492+0000", 2023-11-01T04:16:22.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.957494+0000", 2023-11-01T04:16:22.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4430000000000002e-06 2023-11-01T04:16:22.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.958130+0000", 2023-11-01T04:16:22.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00063580600000000004 2023-11-01T04:16:22.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.958162+0000", 2023-11-01T04:16:22.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2670999999999999e-05 2023-11-01T04:16:22.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.958179+0000", 2023-11-01T04:16:22.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6631999999999999e-05 2023-11-01T04:16:22.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.247961+0000", 2023-11-01T04:16:22.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28978161600000002 2023-11-01T04:16:22.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:22.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.248112+0000", 2023-11-01T04:16:22.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000151781 2023-11-01T04:16:22.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:22.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.250243+0000", 2023-11-01T04:16:22.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021303839999999999 2023-11-01T04:16:22.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:22.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.258223+0000", 2023-11-01T04:16:22.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0079804449999999992 2023-11-01T04:16:22.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:22.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.577520+0000", 2023-11-01T04:16:22.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31929720700000003 2023-11-01T04:16:22.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.577636+0000", 2023-11-01T04:16:22.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000115265 2023-11-01T04:16:22.626 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.627 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.627 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.627 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:182 3.9s0 3:959ff18f:::smithi19039307-32:head [append 0~376832] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:22.628 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.963714+0000", 2023-11-01T04:16:22.628 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.5884983940000001, 2023-11-01T04:16:22.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.61422031099999996, 2023-11-01T04:16:22.628 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.628 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:22.628 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:22.629 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:22.629 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:22.629 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 182 2023-11-01T04:16:22.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.629 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963714+0000", 2023-11-01T04:16:22.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963714+0000", 2023-11-01T04:16:22.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963716+0000", 2023-11-01T04:16:22.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3099999999999999e-06 2023-11-01T04:16:22.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964289+0000", 2023-11-01T04:16:22.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00057335299999999997 2023-11-01T04:16:22.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964323+0000", 2023-11-01T04:16:22.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3268000000000001e-05 2023-11-01T04:16:22.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964339+0000", 2023-11-01T04:16:22.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6782e-05 2023-11-01T04:16:22.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.252517+0000", 2023-11-01T04:16:22.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28817765499999998 2023-11-01T04:16:22.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:22.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.252667+0000", 2023-11-01T04:16:22.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015000300000000001 2023-11-01T04:16:22.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:22.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.254483+0000", 2023-11-01T04:16:22.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018163249999999999 2023-11-01T04:16:22.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:22.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.258226+0000", 2023-11-01T04:16:22.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037425750000000002 2023-11-01T04:16:22.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:22.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.577851+0000", 2023-11-01T04:16:22.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31962504200000003 2023-11-01T04:16:22.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.577934+0000", 2023-11-01T04:16:22.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2997999999999994e-05 2023-11-01T04:16:22.641 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.641 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.642 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.642 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:183 3.9s0 3:959ff18f:::smithi19039307-32:head [setxattr _header (56) in=7b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:22.642 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.964367+0000", 2023-11-01T04:16:22.643 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.5878452300000001, 2023-11-01T04:16:22.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.61393845700000005, 2023-11-01T04:16:22.643 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.643 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:22.643 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:22.644 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:22.644 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:22.644 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 183 2023-11-01T04:16:22.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.644 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964367+0000", 2023-11-01T04:16:22.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964367+0000", 2023-11-01T04:16:22.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964368+0000", 2023-11-01T04:16:22.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3039999999999999e-06 2023-11-01T04:16:22.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964380+0000", 2023-11-01T04:16:22.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1905999999999999e-05 2023-11-01T04:16:22.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964392+0000", 2023-11-01T04:16:22.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1769e-05 2023-11-01T04:16:22.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964401+0000", 2023-11-01T04:16:22.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9260000000000006e-06 2023-11-01T04:16:22.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.255768+0000", 2023-11-01T04:16:22.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.291367502 2023-11-01T04:16:22.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:22.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.256021+0000", 2023-11-01T04:16:22.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025249400000000003 2023-11-01T04:16:22.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.257165+0000", 2023-11-01T04:16:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00114455 2023-11-01T04:16:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.358396+0000", 2023-11-01T04:16:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101230256 2023-11-01T04:16:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.578140+0000", 2023-11-01T04:16:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21974475700000001 2023-11-01T04:16:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.578305+0000", 2023-11-01T04:16:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000164993 2023-11-01T04:16:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:184 3.9s0 3:959ff18f:::smithi19039307-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.964414+0000", 2023-11-01T04:16:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.5877984989999998, 2023-11-01T04:16:22.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.72156181200000002, 2023-11-01T04:16:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:22.658 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 184 2023-11-01T04:16:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964414+0000", 2023-11-01T04:16:22.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964414+0000", 2023-11-01T04:16:22.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964414+0000", 2023-11-01T04:16:22.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2600000000000001e-07 2023-11-01T04:16:22.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964418+0000", 2023-11-01T04:16:22.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5899999999999999e-06 2023-11-01T04:16:22.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964429+0000", 2023-11-01T04:16:22.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0616e-05 2023-11-01T04:16:22.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964437+0000", 2023-11-01T04:16:22.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9470000000000003e-06 2023-11-01T04:16:22.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.259490+0000", 2023-11-01T04:16:22.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29505374699999998 2023-11-01T04:16:22.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for rw locks", 2023-11-01T04:16:22.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.259647+0000", 2023-11-01T04:16:22.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015683599999999999 2023-11-01T04:16:22.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.579080+0000", 2023-11-01T04:16:22.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31943243399999999 2023-11-01T04:16:22.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:22.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.579233+0000", 2023-11-01T04:16:22.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000153621 2023-11-01T04:16:22.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.685975+0000", 2023-11-01T04:16:22.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.106742195 2023-11-01T04:16:22.670 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.670 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.670 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.671 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.9s0 36/31 ECSubReadReply(tid=11, attrs_read=0))", 2023-11-01T04:16:22.671 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.043021+0000", 2023-11-01T04:16:22.671 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.5091906560000004, 2023-11-01T04:16:22.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20414104399999999, 2023-11-01T04:16:22.672 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.672 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.672 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.043021+0000", 2023-11-01T04:16:22.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.043021+0000", 2023-11-01T04:16:22.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.043023+0000", 2023-11-01T04:16:22.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.426e-06 2023-11-01T04:16:22.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.043293+0000", 2023-11-01T04:16:22.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026972299999999998 2023-11-01T04:16:22.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.043324+0000", 2023-11-01T04:16:22.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1100999999999999e-05 2023-11-01T04:16:22.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.043343+0000", 2023-11-01T04:16:22.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9116000000000001e-05 2023-11-01T04:16:22.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.246427+0000", 2023-11-01T04:16:22.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.203084139 2023-11-01T04:16:22.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.247162+0000", 2023-11-01T04:16:22.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00073553900000000001 2023-11-01T04:16:22.680 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.680 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.680 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.681 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.9s0 37/31 ECSubReadReply(tid=12, attrs_read=0))", 2023-11-01T04:16:22.681 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.044307+0000", 2023-11-01T04:16:22.681 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.5079052879999999, 2023-11-01T04:16:22.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.203567154, 2023-11-01T04:16:22.682 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.682 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.682 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.044307+0000", 2023-11-01T04:16:22.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.044307+0000", 2023-11-01T04:16:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.044308+0000", 2023-11-01T04:16:22.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.11e-06 2023-11-01T04:16:22.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.044525+0000", 2023-11-01T04:16:22.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021665599999999999 2023-11-01T04:16:22.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.044546+0000", 2023-11-01T04:16:22.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1375000000000001e-05 2023-11-01T04:16:22.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.044556+0000", 2023-11-01T04:16:22.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0210000000000001e-05 2023-11-01T04:16:22.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.247229+0000", 2023-11-01T04:16:22.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20267279099999999 2023-11-01T04:16:22.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.247874+0000", 2023-11-01T04:16:22.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064501200000000004 2023-11-01T04:16:22.690 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.690 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.690 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s1 41/31 ECSubWrite(tid=16, reqid=osd.3.0:15, at_version=0'0, trim_to=0'0, roll_forward_to=41'10))", 2023-11-01T04:16:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.435629+0000", 2023-11-01T04:16:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.1165830809999999, 2023-11-01T04:16:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20179989100000001, 2023-11-01T04:16:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.435629+0000", 2023-11-01T04:16:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.435629+0000", 2023-11-01T04:16:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.435630+0000", 2023-11-01T04:16:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9899999999999999e-07 2023-11-01T04:16:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.435647+0000", 2023-11-01T04:16:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7464000000000001e-05 2023-11-01T04:16:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.435664+0000", 2023-11-01T04:16:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6531000000000001e-05 2023-11-01T04:16:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.535798+0000", 2023-11-01T04:16:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100133607 2023-11-01T04:16:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.636073+0000", 2023-11-01T04:16:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100275009 2023-11-01T04:16:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.636114+0000", 2023-11-01T04:16:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1411e-05 2023-11-01T04:16:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.637379+0000", 2023-11-01T04:16:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012654649999999999 2023-11-01T04:16:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.637429+0000", 2023-11-01T04:16:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9505000000000001e-05 2023-11-01T04:16:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 41/31 ECSubWriteReply(tid=29, last_complete=41'22, committed=1, applied=1))", 2023-11-01T04:16:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.446839+0000", 2023-11-01T04:16:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.1053728839999999, 2023-11-01T04:16:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29203878100000003, 2023-11-01T04:16:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.446839+0000", 2023-11-01T04:16:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.446839+0000", 2023-11-01T04:16:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.446840+0000", 2023-11-01T04:16:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.127e-06 2023-11-01T04:16:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.446847+0000", 2023-11-01T04:16:22.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1789999999999996e-06 2023-11-01T04:16:22.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.446864+0000", 2023-11-01T04:16:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7368999999999999e-05 2023-11-01T04:16:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.446873+0000", 2023-11-01T04:16:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4099999999999997e-06 2023-11-01T04:16:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.738847+0000", 2023-11-01T04:16:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29197415500000001 2023-11-01T04:16:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.738878+0000", 2023-11-01T04:16:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0540999999999998e-05 2023-11-01T04:16:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 41/31 ECSubWriteReply(tid=30, last_complete=41'24, committed=1, applied=1))", 2023-11-01T04:16:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.447401+0000", 2023-11-01T04:16:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.1048109850000001, 2023-11-01T04:16:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29306723299999998, 2023-11-01T04:16:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.447401+0000", 2023-11-01T04:16:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.447401+0000", 2023-11-01T04:16:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.447402+0000", 2023-11-01T04:16:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.47e-07 2023-11-01T04:16:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.447407+0000", 2023-11-01T04:16:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6940000000000003e-06 2023-11-01T04:16:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.447420+0000", 2023-11-01T04:16:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2884000000000001e-05 2023-11-01T04:16:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.447427+0000", 2023-11-01T04:16:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0140000000000001e-06 2023-11-01T04:16:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.739182+0000", 2023-11-01T04:16:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29175571 2023-11-01T04:16:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.740468+0000", 2023-11-01T04:16:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012860840000000001 2023-11-01T04:16:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 41/31 ECSubWriteReply(tid=29, last_complete=41'22, committed=1, applied=1))", 2023-11-01T04:16:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.448033+0000", 2023-11-01T04:16:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.104178753, 2023-11-01T04:16:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29059283800000002, 2023-11-01T04:16:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.448033+0000", 2023-11-01T04:16:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.448033+0000", 2023-11-01T04:16:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.448034+0000", 2023-11-01T04:16:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9100000000000003e-07 2023-11-01T04:16:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.448038+0000", 2023-11-01T04:16:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7610000000000001e-06 2023-11-01T04:16:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.448049+0000", 2023-11-01T04:16:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1664e-05 2023-11-01T04:16:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.448055+0000", 2023-11-01T04:16:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6949999999999997e-06 2023-11-01T04:16:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.738550+0000", 2023-11-01T04:16:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29049473799999997 2023-11-01T04:16:22.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.738626+0000", 2023-11-01T04:16:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6289000000000001e-05 2023-11-01T04:16:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 41/31 ECSubWriteReply(tid=30, last_complete=41'24, committed=1, applied=1))", 2023-11-01T04:16:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.448334+0000", 2023-11-01T04:16:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.1038778220000001, 2023-11-01T04:16:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29062167799999999, 2023-11-01T04:16:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.448334+0000", 2023-11-01T04:16:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.448334+0000", 2023-11-01T04:16:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.448335+0000", 2023-11-01T04:16:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.47e-07 2023-11-01T04:16:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.448340+0000", 2023-11-01T04:16:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5430000000000004e-06 2023-11-01T04:16:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.448355+0000", 2023-11-01T04:16:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5707999999999999e-05 2023-11-01T04:16:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.448366+0000", 2023-11-01T04:16:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0778000000000001e-05 2023-11-01T04:16:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.738926+0000", 2023-11-01T04:16:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29056026400000001 2023-11-01T04:16:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.738956+0000", 2023-11-01T04:16:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9437999999999998e-05 2023-11-01T04:16:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 41/31 ECSubWriteReply(tid=30, last_complete=41'24, committed=1, applied=1))", 2023-11-01T04:16:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.449495+0000", 2023-11-01T04:16:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.1027167630000001, 2023-11-01T04:16:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.289533918, 2023-11-01T04:16:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.449495+0000", 2023-11-01T04:16:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.449495+0000", 2023-11-01T04:16:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.449496+0000", 2023-11-01T04:16:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7000000000000005e-07 2023-11-01T04:16:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.449499+0000", 2023-11-01T04:16:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1980000000000001e-06 2023-11-01T04:16:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.449510+0000", 2023-11-01T04:16:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0522e-05 2023-11-01T04:16:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.449515+0000", 2023-11-01T04:16:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1660000000000002e-06 2023-11-01T04:16:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.739010+0000", 2023-11-01T04:16:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.289494959 2023-11-01T04:16:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.739029+0000", 2023-11-01T04:16:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9502999999999999e-05 2023-11-01T04:16:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 41/31 ECSubWriteReply(tid=29, last_complete=41'22, committed=1, applied=1))", 2023-11-01T04:16:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.537584+0000", 2023-11-01T04:16:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.014628149, 2023-11-01T04:16:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20113525600000001, 2023-11-01T04:16:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.753 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.753 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.753 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.753 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.537584+0000", 2023-11-01T04:16:22.753 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.537584+0000", 2023-11-01T04:16:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.537585+0000", 2023-11-01T04:16:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.144e-06 2023-11-01T04:16:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.537590+0000", 2023-11-01T04:16:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0089999999999998e-06 2023-11-01T04:16:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.537602+0000", 2023-11-01T04:16:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1772e-05 2023-11-01T04:16:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.537608+0000", 2023-11-01T04:16:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5130000000000002e-06 2023-11-01T04:16:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.738698+0000", 2023-11-01T04:16:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20108931499999999 2023-11-01T04:16:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.738719+0000", 2023-11-01T04:16:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1503000000000001e-05 2023-11-01T04:16:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 41/31 ECSubWriteReply(tid=29, last_complete=41'22, committed=1, applied=1))", 2023-11-01T04:16:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.537981+0000", 2023-11-01T04:16:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.0142310160000001, 2023-11-01T04:16:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20082561099999999, 2023-11-01T04:16:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.537981+0000", 2023-11-01T04:16:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.537981+0000", 2023-11-01T04:16:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.537981+0000", 2023-11-01T04:16:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2000000000000001e-07 2023-11-01T04:16:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.537984+0000", 2023-11-01T04:16:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3020000000000002e-06 2023-11-01T04:16:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.537995+0000", 2023-11-01T04:16:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1263999999999999e-05 2023-11-01T04:16:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.538002+0000", 2023-11-01T04:16:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9310000000000002e-06 2023-11-01T04:16:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.738776+0000", 2023-11-01T04:16:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20077398599999999 2023-11-01T04:16:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.738807+0000", 2023-11-01T04:16:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0808e-05 2023-11-01T04:16:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 41/31 ECSubWriteReply(tid=30, last_complete=41'24, committed=1, applied=1))", 2023-11-01T04:16:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.538090+0000", 2023-11-01T04:16:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.014122448, 2023-11-01T04:16:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20101334600000001, 2023-11-01T04:16:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.538090+0000", 2023-11-01T04:16:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.538090+0000", 2023-11-01T04:16:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.538090+0000", 2023-11-01T04:16:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.06e-07 2023-11-01T04:16:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.538093+0000", 2023-11-01T04:16:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8959999999999999e-06 2023-11-01T04:16:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.538101+0000", 2023-11-01T04:16:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.622e-06 2023-11-01T04:16:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.538108+0000", 2023-11-01T04:16:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1609999999999997e-06 2023-11-01T04:16:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.739075+0000", 2023-11-01T04:16:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20096730300000001 2023-11-01T04:16:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.739103+0000", 2023-11-01T04:16:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7858e-05 2023-11-01T04:16:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:22.782 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:22.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-01T04:16:22.784 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:22.782+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:22.884 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:22.882+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:22.897 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-01T04:16:22.912 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:22.912 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:22.912 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2023-11-01T04:16:22.913 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:22.927 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-01T04:16:22.985 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:22.982+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:22.998 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.3 to 1 (10000) 2023-11-01T04:16:22.999 INFO:teuthology.orchestra.run.smithi153.stderr:nodeep-scrub is set 2023-11-01T04:16:23.008 INFO:tasks.rados.rados.0.smithi190.stdout:102: done (0 left) 2023-11-01T04:16:23.008 INFO:tasks.rados.rados.0.smithi190.stdout:103: read oid 29 snap 4 2023-11-01T04:16:23.008 INFO:tasks.rados.rados.0.smithi190.stdout:103: expect deleted 2023-11-01T04:16:23.009 INFO:tasks.rados.rados.0.smithi190.stdout:104: read oid 46 snap -1 2023-11-01T04:16:23.009 INFO:tasks.rados.rados.0.smithi190.stdout:104: expect (ObjNum 45 snap 0 seq_num 45) 2023-11-01T04:16:23.009 INFO:tasks.rados.rados.0.smithi190.stdout:105: copy_from oid 36 from oid 2 current snap is 5 2023-11-01T04:16:23.009 INFO:tasks.rados.rados.0.smithi190.stdout:106: rmattr oid 49 current snap is 5 2023-11-01T04:16:23.009 INFO:tasks.rados.rados.0.smithi190.stdout:106: done (3 left) 2023-11-01T04:16:23.010 INFO:tasks.rados.rados.0.smithi190.stdout:107: snap_remove snap 3 2023-11-01T04:16:23.025 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2023-11-01T04:16:23.046 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:23.047 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:23.047 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2023-11-01T04:16:23.047 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2023-11-01T04:16:23.047 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:23.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-01T04:16:23.085 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:23.086+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:23.139 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:23.140 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-11-01T04:16:23.167 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:23.167 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-01T04:16:23.186 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:23.183+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:23.190 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:23.190 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2023-11-01T04:16:23.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2023-11-01T04:16:23.191 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2023-11-01T04:16:23.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.191 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:185 3.es0 3:723efae4:::smithi19039307-36:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:23.191 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.963781+0000", 2023-11-01T04:16:23.192 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.22532745, 2023-11-01T04:16:23.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31012846799999999, 2023-11-01T04:16:23.192 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.192 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:23.192 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:23.193 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:23.193 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:23.193 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185 2023-11-01T04:16:23.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.194 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963781+0000", 2023-11-01T04:16:23.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.195 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963781+0000", 2023-11-01T04:16:23.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.196 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.196 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963785+0000", 2023-11-01T04:16:23.197 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.563e-06 2023-11-01T04:16:23.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.197 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.197 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963797+0000", 2023-11-01T04:16:23.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2201999999999999e-05 2023-11-01T04:16:23.198 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.198 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963839+0000", 2023-11-01T04:16:23.199 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2181000000000002e-05 2023-11-01T04:16:23.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.199 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963864+0000", 2023-11-01T04:16:23.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4419000000000001e-05 2023-11-01T04:16:23.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.200 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.200 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964036+0000", 2023-11-01T04:16:23.201 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017209500000000001 2023-11-01T04:16:23.201 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:23.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964291+0000", 2023-11-01T04:16:23.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025550699999999998 2023-11-01T04:16:23.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.202 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.202 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:23.202 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966403+0000", 2023-11-01T04:16:23.203 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021113500000000001 2023-11-01T04:16:23.203 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:23.204 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.072479+0000", 2023-11-01T04:16:23.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10607620800000001 2023-11-01T04:16:23.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.204 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:23.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.273808+0000", 2023-11-01T04:16:23.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20132876299999999 2023-11-01T04:16:23.205 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.205 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.206 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.273910+0000", 2023-11-01T04:16:23.206 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010218 2023-11-01T04:16:23.206 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.206 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.206 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.207 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.207 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.207 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:186 3.es0 3:723efae4:::smithi19039307-36:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:23.207 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.963910+0000", 2023-11-01T04:16:23.207 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.2251989050000001, 2023-11-01T04:16:23.208 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.410473274, 2023-11-01T04:16:23.208 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.208 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:23.208 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:23.209 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:23.209 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:23.209 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 186 2023-11-01T04:16:23.209 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.209 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.210 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.210 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.210 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963910+0000", 2023-11-01T04:16:23.210 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.210 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.211 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.211 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963910+0000", 2023-11-01T04:16:23.211 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.212 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.212 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.212 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963911+0000", 2023-11-01T04:16:23.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.28e-06 2023-11-01T04:16:23.212 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.213 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.213 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963917+0000", 2023-11-01T04:16:23.213 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1829999999999999e-06 2023-11-01T04:16:23.213 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.213 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.214 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.214 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963942+0000", 2023-11-01T04:16:23.214 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4808e-05 2023-11-01T04:16:23.214 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.214 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.215 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.215 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963957+0000", 2023-11-01T04:16:23.215 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4698999999999999e-05 2023-11-01T04:16:23.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.215 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.967838+0000", 2023-11-01T04:16:23.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003881139 2023-11-01T04:16:23.216 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:23.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.968035+0000", 2023-11-01T04:16:23.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019704400000000001 2023-11-01T04:16:23.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:23.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.968889+0000", 2023-11-01T04:16:23.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00085385099999999998 2023-11-01T04:16:23.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:23.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.072489+0000", 2023-11-01T04:16:23.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.103600028 2023-11-01T04:16:23.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:23.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374278+0000", 2023-11-01T04:16:23.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.301788789 2023-11-01T04:16:23.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374383+0000", 2023-11-01T04:16:23.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010545299999999999 2023-11-01T04:16:23.221 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.221 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.221 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.222 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:187 3.es0 3:723efae4:::smithi19039307-36:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:23.222 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.964019+0000", 2023-11-01T04:16:23.222 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.225090142, 2023-11-01T04:16:23.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41093530900000003, 2023-11-01T04:16:23.223 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.223 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:23.223 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:23.223 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:23.224 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:23.224 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 187 2023-11-01T04:16:23.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.224 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964019+0000", 2023-11-01T04:16:23.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964019+0000", 2023-11-01T04:16:23.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964020+0000", 2023-11-01T04:16:23.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.26e-06 2023-11-01T04:16:23.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964026+0000", 2023-11-01T04:16:23.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3260000000000001e-06 2023-11-01T04:16:23.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964049+0000", 2023-11-01T04:16:23.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2255e-05 2023-11-01T04:16:23.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964067+0000", 2023-11-01T04:16:23.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8643000000000001e-05 2023-11-01T04:16:23.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.969573+0000", 2023-11-01T04:16:23.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055055729999999997 2023-11-01T04:16:23.231 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.231 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.231 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:23.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.969770+0000", 2023-11-01T04:16:23.232 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000196787 2023-11-01T04:16:23.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:23.232 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.971048+0000", 2023-11-01T04:16:23.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012779709999999999 2023-11-01T04:16:23.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.233 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.233 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:23.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.072491+0000", 2023-11-01T04:16:23.234 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101443751 2023-11-01T04:16:23.234 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:23.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374869+0000", 2023-11-01T04:16:23.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.302377644 2023-11-01T04:16:23.235 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.235 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.235 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.235 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374954+0000", 2023-11-01T04:16:23.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5099000000000006e-05 2023-11-01T04:16:23.236 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.236 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.236 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.236 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.237 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:194 3.es0 3:706dddd5:::smithi19039307-14:head [copy-from ver 6] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:23.237 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966030+0000", 2023-11-01T04:16:23.237 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.2230786089999999, 2023-11-01T04:16:23.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.72279825099999995, 2023-11-01T04:16:23.237 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.238 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:23.238 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:23.238 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:23.238 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:23.238 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 194 2023-11-01T04:16:23.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.239 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.239 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.239 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.239 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966030+0000", 2023-11-01T04:16:23.239 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.240 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966030+0000", 2023-11-01T04:16:23.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.241 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.241 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.241 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966032+0000", 2023-11-01T04:16:23.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9470000000000002e-06 2023-11-01T04:16:23.242 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.242 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966043+0000", 2023-11-01T04:16:23.242 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0396e-05 2023-11-01T04:16:23.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.243 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.243 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966073+0000", 2023-11-01T04:16:23.244 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0518000000000002e-05 2023-11-01T04:16:23.244 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966093+0000", 2023-11-01T04:16:23.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9955000000000001e-05 2023-11-01T04:16:23.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.245 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.071960+0000", 2023-11-01T04:16:23.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.105867171 2023-11-01T04:16:23.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.246 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.246 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:23.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.072114+0000", 2023-11-01T04:16:23.247 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000153323 2023-11-01T04:16:23.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:23.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.480839+0000", 2023-11-01T04:16:23.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40872525500000001 2023-11-01T04:16:23.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.248 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:23.248 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.487356+0000", 2023-11-01T04:16:23.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0065175770000000001 2023-11-01T04:16:23.249 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.249 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:23.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.688636+0000", 2023-11-01T04:16:23.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20127930899999999 2023-11-01T04:16:23.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.250 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.688829+0000", 2023-11-01T04:16:23.251 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0001928 2023-11-01T04:16:23.251 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.251 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.251 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.252 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.252 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:195 3.es0 3:706dddd5:::smithi19039307-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:23.252 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966129+0000", 2023-11-01T04:16:23.252 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.2229803190000004, 2023-11-01T04:16:23.252 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51772273199999996, 2023-11-01T04:16:23.252 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.253 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:23.253 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:23.253 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:23.253 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:23.253 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 195 2023-11-01T04:16:23.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.254 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966129+0000", 2023-11-01T04:16:23.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966129+0000", 2023-11-01T04:16:23.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.256 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.256 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966130+0000", 2023-11-01T04:16:23.256 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4529999999999999e-06 2023-11-01T04:16:23.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.257 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.257 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.257 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966168+0000", 2023-11-01T04:16:23.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7792e-05 2023-11-01T04:16:23.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.258 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.258 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966192+0000", 2023-11-01T04:16:23.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4083999999999998e-05 2023-11-01T04:16:23.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966210+0000", 2023-11-01T04:16:23.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8196e-05 2023-11-01T04:16:23.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.173008+0000", 2023-11-01T04:16:23.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20679805200000001 2023-11-01T04:16:23.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2023-11-01T04:16:23.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.173168+0000", 2023-11-01T04:16:23.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000159457 2023-11-01T04:16:23.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.483564+0000", 2023-11-01T04:16:23.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31039652000000001 2023-11-01T04:16:23.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:23.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.483711+0000", 2023-11-01T04:16:23.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014698099999999999 2023-11-01T04:16:23.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.483851+0000", 2023-11-01T04:16:23.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000140197 2023-11-01T04:16:23.264 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.265 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.265 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.265 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:202 3.es0 3:7f8a06cd:::smithi19039307-49:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:23.266 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966420+0000", 2023-11-01T04:16:23.266 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.2226890109999999, 2023-11-01T04:16:23.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.91933000499999995, 2023-11-01T04:16:23.266 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.266 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:23.267 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:23.267 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:23.267 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:23.267 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 202 2023-11-01T04:16:23.267 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.268 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966420+0000", 2023-11-01T04:16:23.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.269 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.269 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966420+0000", 2023-11-01T04:16:23.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.270 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.270 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966421+0000", 2023-11-01T04:16:23.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6199999999999999e-06 2023-11-01T04:16:23.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966431+0000", 2023-11-01T04:16:23.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4660000000000005e-06 2023-11-01T04:16:23.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966456+0000", 2023-11-01T04:16:23.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4748000000000001e-05 2023-11-01T04:16:23.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966473+0000", 2023-11-01T04:16:23.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7323e-05 2023-11-01T04:16:23.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.375871+0000", 2023-11-01T04:16:23.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40939844800000003 2023-11-01T04:16:23.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:23.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.376008+0000", 2023-11-01T04:16:23.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000136997 2023-11-01T04:16:23.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.276 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:23.276 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.585026+0000", 2023-11-01T04:16:23.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.209017436 2023-11-01T04:16:23.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:23.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.688063+0000", 2023-11-01T04:16:23.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.103037031 2023-11-01T04:16:23.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:23.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.885528+0000", 2023-11-01T04:16:23.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.19746507999999999 2023-11-01T04:16:23.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.885750+0000", 2023-11-01T04:16:23.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022185600000000001 2023-11-01T04:16:23.279 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.279 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.280 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.280 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:203 3.es0 3:7f8a06cd:::smithi19039307-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:23.280 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966500+0000", 2023-11-01T04:16:23.281 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.2226086130000002, 2023-11-01T04:16:23.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.62108323700000001, 2023-11-01T04:16:23.281 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.281 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:23.281 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:23.282 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:23.282 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:23.282 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 203 2023-11-01T04:16:23.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.282 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.283 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.283 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.283 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966500+0000", 2023-11-01T04:16:23.283 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.284 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.284 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966500+0000", 2023-11-01T04:16:23.284 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.284 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.284 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.285 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.285 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966501+0000", 2023-11-01T04:16:23.285 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1570000000000001e-06 2023-11-01T04:16:23.285 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.285 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.286 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.286 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966530+0000", 2023-11-01T04:16:23.286 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8337999999999999e-05 2023-11-01T04:16:23.286 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.286 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.287 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966549+0000", 2023-11-01T04:16:23.287 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9624999999999999e-05 2023-11-01T04:16:23.287 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.287 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.287 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.288 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966584+0000", 2023-11-01T04:16:23.288 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4442e-05 2023-11-01T04:16:23.288 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.288 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.288 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.289 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.376334+0000", 2023-11-01T04:16:23.289 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.409750269 2023-11-01T04:16:23.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.289 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.289 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2023-11-01T04:16:23.290 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.376468+0000", 2023-11-01T04:16:23.290 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000133588 2023-11-01T04:16:23.290 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.290 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.290 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.586951+0000", 2023-11-01T04:16:23.291 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21048317499999999 2023-11-01T04:16:23.291 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.291 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.291 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:23.291 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.587248+0000", 2023-11-01T04:16:23.292 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029743400000000001 2023-11-01T04:16:23.292 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.292 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.292 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.293 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.587583+0000", 2023-11-01T04:16:23.293 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000335209 2023-11-01T04:16:23.293 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.293 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.293 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.294 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.294 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=22, last_complete=37'15, committed=1, applied=1))", 2023-11-01T04:16:23.294 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.073048+0000", 2023-11-01T04:16:23.294 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.1160607049999998, 2023-11-01T04:16:23.295 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30149571800000002, 2023-11-01T04:16:23.295 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.295 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:23.295 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.295 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.296 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.296 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073048+0000", 2023-11-01T04:16:23.296 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.296 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.296 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.297 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.297 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073048+0000", 2023-11-01T04:16:23.297 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.297 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.297 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.298 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.298 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073049+0000", 2023-11-01T04:16:23.298 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.96e-07 2023-11-01T04:16:23.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.299 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.299 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.299 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073052+0000", 2023-11-01T04:16:23.299 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9739999999999998e-06 2023-11-01T04:16:23.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.300 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.300 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.300 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073061+0000", 2023-11-01T04:16:23.300 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.003e-06 2023-11-01T04:16:23.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.301 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.301 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.301 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073067+0000", 2023-11-01T04:16:23.301 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6560000000000003e-06 2023-11-01T04:16:23.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.302 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.302 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.302 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374508+0000", 2023-11-01T04:16:23.302 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30144055400000003 2023-11-01T04:16:23.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.303 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374544+0000", 2023-11-01T04:16:23.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6134999999999999e-05 2023-11-01T04:16:23.304 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.304 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.304 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.305 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=21, last_complete=37'14, committed=1, applied=1))", 2023-11-01T04:16:23.305 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.073145+0000", 2023-11-01T04:16:23.305 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.1159634909999996, 2023-11-01T04:16:23.306 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30125827300000002, 2023-11-01T04:16:23.306 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.306 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:23.306 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073145+0000", 2023-11-01T04:16:23.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.308 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073145+0000", 2023-11-01T04:16:23.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073146+0000", 2023-11-01T04:16:23.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8599999999999999e-07 2023-11-01T04:16:23.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.310 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073149+0000", 2023-11-01T04:16:23.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8339999999999999e-06 2023-11-01T04:16:23.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073160+0000", 2023-11-01T04:16:23.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1671999999999999e-05 2023-11-01T04:16:23.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073168+0000", 2023-11-01T04:16:23.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2269999999999996e-06 2023-11-01T04:16:23.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374161+0000", 2023-11-01T04:16:23.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30099383499999999 2023-11-01T04:16:23.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.314 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374404+0000", 2023-11-01T04:16:23.314 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024231900000000001 2023-11-01T04:16:23.314 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.314 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.314 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.315 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.315 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=22, last_complete=37'15, committed=1, applied=1))", 2023-11-01T04:16:23.315 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.073177+0000", 2023-11-01T04:16:23.315 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.1159315599999999, 2023-11-01T04:16:23.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30259569400000003, 2023-11-01T04:16:23.316 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.316 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:23.316 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.316 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.316 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073177+0000", 2023-11-01T04:16:23.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073177+0000", 2023-11-01T04:16:23.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073178+0000", 2023-11-01T04:16:23.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6300000000000001e-07 2023-11-01T04:16:23.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.319 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.320 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073180+0000", 2023-11-01T04:16:23.320 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.65e-06 2023-11-01T04:16:23.320 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.320 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.321 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.321 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073188+0000", 2023-11-01T04:16:23.321 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0730000000000005e-06 2023-11-01T04:16:23.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.322 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073194+0000", 2023-11-01T04:16:23.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1800000000000001e-06 2023-11-01T04:16:23.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.323 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374769+0000", 2023-11-01T04:16:23.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30157417600000003 2023-11-01T04:16:23.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.375773+0000", 2023-11-01T04:16:23.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001004352 2023-11-01T04:16:23.324 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.324 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.325 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.325 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=22, last_complete=37'15, committed=1, applied=1))", 2023-11-01T04:16:23.325 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.075657+0000", 2023-11-01T04:16:23.325 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.1134516510000001, 2023-11-01T04:16:23.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29898018399999998, 2023-11-01T04:16:23.326 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.326 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:23.326 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.075657+0000", 2023-11-01T04:16:23.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.327 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.075657+0000", 2023-11-01T04:16:23.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.075659+0000", 2023-11-01T04:16:23.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5829999999999999e-06 2023-11-01T04:16:23.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.330 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.075668+0000", 2023-11-01T04:16:23.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9889999999999998e-06 2023-11-01T04:16:23.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.075696+0000", 2023-11-01T04:16:23.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8218e-05 2023-11-01T04:16:23.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.075712+0000", 2023-11-01T04:16:23.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5736e-05 2023-11-01T04:16:23.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374611+0000", 2023-11-01T04:16:23.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29889970700000001 2023-11-01T04:16:23.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374637+0000", 2023-11-01T04:16:23.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5950999999999999e-05 2023-11-01T04:16:23.334 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.334 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.334 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.335 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=21, last_complete=37'14, committed=1, applied=1))", 2023-11-01T04:16:23.335 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.171098+0000", 2023-11-01T04:16:23.335 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.0180108299999997, 2023-11-01T04:16:23.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20302295300000001, 2023-11-01T04:16:23.336 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.336 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:23.336 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.336 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.337 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.171098+0000", 2023-11-01T04:16:23.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.337 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.337 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.338 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.171098+0000", 2023-11-01T04:16:23.338 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.338 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.338 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.171099+0000", 2023-11-01T04:16:23.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.083e-06 2023-11-01T04:16:23.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.171107+0000", 2023-11-01T04:16:23.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4830000000000002e-06 2023-11-01T04:16:23.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.341 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.171128+0000", 2023-11-01T04:16:23.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1124999999999998e-05 2023-11-01T04:16:23.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.171139+0000", 2023-11-01T04:16:23.342 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1483e-05 2023-11-01T04:16:23.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.342 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.342 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374092+0000", 2023-11-01T04:16:23.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20295264800000001 2023-11-01T04:16:23.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374121+0000", 2023-11-01T04:16:23.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9130999999999999e-05 2023-11-01T04:16:23.344 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.344 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.344 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.345 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=22, last_complete=37'15, committed=1, applied=1))", 2023-11-01T04:16:23.345 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.171159+0000", 2023-11-01T04:16:23.345 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.0179499669999998, 2023-11-01T04:16:23.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.203569571, 2023-11-01T04:16:23.346 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.346 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:23.346 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.171159+0000", 2023-11-01T04:16:23.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.171159+0000", 2023-11-01T04:16:23.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.171160+0000", 2023-11-01T04:16:23.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3300000000000001e-07 2023-11-01T04:16:23.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.171164+0000", 2023-11-01T04:16:23.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.843e-06 2023-11-01T04:16:23.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.171181+0000", 2023-11-01T04:16:23.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6796e-05 2023-11-01T04:16:23.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.171194+0000", 2023-11-01T04:16:23.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2741999999999999e-05 2023-11-01T04:16:23.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374697+0000", 2023-11-01T04:16:23.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.203502559 2023-11-01T04:16:23.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374728+0000", 2023-11-01T04:16:23.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1897999999999999e-05 2023-11-01T04:16:23.354 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.354 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.354 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.355 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=24, last_complete=37'17, committed=1, applied=1))", 2023-11-01T04:16:23.355 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.484201+0000", 2023-11-01T04:16:23.355 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.7049077690000001, 2023-11-01T04:16:23.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20359449700000001, 2023-11-01T04:16:23.356 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.356 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:23.356 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.484201+0000", 2023-11-01T04:16:23.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.484201+0000", 2023-11-01T04:16:23.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.484209+0000", 2023-11-01T04:16:23.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8229999999999995e-06 2023-11-01T04:16:23.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.484220+0000", 2023-11-01T04:16:23.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1039999999999999e-05 2023-11-01T04:16:23.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.484245+0000", 2023-11-01T04:16:23.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5298000000000001e-05 2023-11-01T04:16:23.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.584332+0000", 2023-11-01T04:16:23.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100086439 2023-11-01T04:16:23.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.687740+0000", 2023-11-01T04:16:23.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.103407998 2023-11-01T04:16:23.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.687796+0000", 2023-11-01T04:16:23.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5899000000000001e-05 2023-11-01T04:16:23.364 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.364 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.364 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.365 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:270 3.es0 3:706dddd5:::smithi19039307-14:head [read 0~638976,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-01T04:16:23.365 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.017359+0000", 2023-11-01T04:16:23.365 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 2.1717501960000001, 2023-11-01T04:16:23.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21120662700000001, 2023-11-01T04:16:23.365 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.366 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:23.366 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:23.366 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:23.366 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:23.366 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 270 2023-11-01T04:16:23.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.367 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017359+0000", 2023-11-01T04:16:23.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017359+0000", 2023-11-01T04:16:23.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017363+0000", 2023-11-01T04:16:23.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6800000000000001e-06 2023-11-01T04:16:23.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017382+0000", 2023-11-01T04:16:23.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8495000000000002e-05 2023-11-01T04:16:23.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017463+0000", 2023-11-01T04:16:23.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1058999999999998e-05 2023-11-01T04:16:23.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017483+0000", 2023-11-01T04:16:23.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9718999999999999e-05 2023-11-01T04:16:23.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017613+0000", 2023-11-01T04:16:23.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000130109 2023-11-01T04:16:23.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2023-11-01T04:16:23.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017624+0000", 2023-11-01T04:16:23.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1389999999999999e-05 2023-11-01T04:16:23.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.021914+0000", 2023-11-01T04:16:23.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042898889999999999 2023-11-01T04:16:23.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:23.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.022163+0000", 2023-11-01T04:16:23.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024884899999999999 2023-11-01T04:16:23.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.228565+0000", 2023-11-01T04:16:23.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20640243699999999 2023-11-01T04:16:23.377 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.377 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.378 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.378 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:271 3.es0 3:706dddd5:::smithi19039307-14:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-01T04:16:23.378 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.017519+0000", 2023-11-01T04:16:23.379 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 2.1715897480000002, 2023-11-01T04:16:23.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31206822200000001, 2023-11-01T04:16:23.379 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.379 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:23.379 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:23.379 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:23.380 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:23.380 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 271 2023-11-01T04:16:23.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.380 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017519+0000", 2023-11-01T04:16:23.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017519+0000", 2023-11-01T04:16:23.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017520+0000", 2023-11-01T04:16:23.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.203e-06 2023-11-01T04:16:23.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017527+0000", 2023-11-01T04:16:23.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7089999999999996e-06 2023-11-01T04:16:23.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017559+0000", 2023-11-01T04:16:23.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1562e-05 2023-11-01T04:16:23.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017568+0000", 2023-11-01T04:16:23.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7189999999999996e-06 2023-11-01T04:16:23.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017754+0000", 2023-11-01T04:16:23.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018541 2023-11-01T04:16:23.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:23.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017764+0000", 2023-11-01T04:16:23.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0609e-05 2023-11-01T04:16:23.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.023462+0000", 2023-11-01T04:16:23.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0056980370000000004 2023-11-01T04:16:23.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:23.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.023706+0000", 2023-11-01T04:16:23.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024344300000000001 2023-11-01T04:16:23.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.329587+0000", 2023-11-01T04:16:23.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30588153000000001 2023-11-01T04:16:23.391 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.391 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.391 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.392 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:272 3.es0 3:706dddd5:::smithi19039307-14:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-01T04:16:23.392 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.017583+0000", 2023-11-01T04:16:23.392 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 2.1715260920000001, 2023-11-01T04:16:23.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.412902926, 2023-11-01T04:16:23.392 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.393 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:23.393 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:23.393 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:23.393 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:23.393 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 272 2023-11-01T04:16:23.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.394 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017583+0000", 2023-11-01T04:16:23.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017583+0000", 2023-11-01T04:16:23.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017584+0000", 2023-11-01T04:16:23.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6000000000000002e-07 2023-11-01T04:16:23.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017588+0000", 2023-11-01T04:16:23.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1330000000000001e-06 2023-11-01T04:16:23.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017598+0000", 2023-11-01T04:16:23.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8430000000000005e-06 2023-11-01T04:16:23.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017605+0000", 2023-11-01T04:16:23.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7030000000000006e-06 2023-11-01T04:16:23.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017822+0000", 2023-11-01T04:16:23.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000217137 2023-11-01T04:16:23.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:23.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017832+0000", 2023-11-01T04:16:23.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2459999999999992e-06 2023-11-01T04:16:23.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.226201+0000", 2023-11-01T04:16:23.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20836893100000001 2023-11-01T04:16:23.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:23.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.226369+0000", 2023-11-01T04:16:23.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016842100000000001 2023-11-01T04:16:23.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.430486+0000", 2023-11-01T04:16:23.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20411665200000001 2023-11-01T04:16:23.405 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.405 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.405 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.405 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s1 41/31 ECSubWrite(tid=36, reqid=osd.14.0:35, at_version=0'0, trim_to=0'0, roll_forward_to=41'14))", 2023-11-01T04:16:23.406 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.537189+0000", 2023-11-01T04:16:23.406 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.6519197779999999, 2023-11-01T04:16:23.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.201429152, 2023-11-01T04:16:23.406 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.406 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:23.407 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.537189+0000", 2023-11-01T04:16:23.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.537189+0000", 2023-11-01T04:16:23.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.537190+0000", 2023-11-01T04:16:23.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2999999999999999e-07 2023-11-01T04:16:23.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.537206+0000", 2023-11-01T04:16:23.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6399e-05 2023-11-01T04:16:23.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.537220+0000", 2023-11-01T04:16:23.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3506000000000001e-05 2023-11-01T04:16:23.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.637325+0000", 2023-11-01T04:16:23.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100104812 2023-11-01T04:16:23.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.737570+0000", 2023-11-01T04:16:23.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10024489 2023-11-01T04:16:23.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:23.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.737580+0000", 2023-11-01T04:16:23.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9979999999999998e-06 2023-11-01T04:16:23.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:23.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.738573+0000", 2023-11-01T04:16:23.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00099342399999999991 2023-11-01T04:16:23.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.738618+0000", 2023-11-01T04:16:23.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5293000000000002e-05 2023-11-01T04:16:23.418 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.418 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.418 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.420 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 41/31 ECSubWriteReply(tid=39, last_complete=41'29, committed=1, applied=1))", 2023-11-01T04:16:23.420 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.852996+0000", 2023-11-01T04:16:23.420 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.3361131070000001, 2023-11-01T04:16:23.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30060843599999998, 2023-11-01T04:16:23.421 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.421 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:23.421 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.852996+0000", 2023-11-01T04:16:23.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.852996+0000", 2023-11-01T04:16:23.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.852997+0000", 2023-11-01T04:16:23.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6599999999999994e-07 2023-11-01T04:16:23.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853007+0000", 2023-11-01T04:16:23.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9429999999999995e-06 2023-11-01T04:16:23.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853024+0000", 2023-11-01T04:16:23.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6889e-05 2023-11-01T04:16:23.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853034+0000", 2023-11-01T04:16:23.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0665999999999999e-05 2023-11-01T04:16:23.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:22.153586+0000", 2023-11-01T04:16:23.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30055170399999998 2023-11-01T04:16:23.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:22.153604+0000", 2023-11-01T04:16:23.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8267999999999999e-05 2023-11-01T04:16:23.429 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.429 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.430 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.430 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 41/31 ECSubWriteReply(tid=39, last_complete=41'29, committed=1, applied=1))", 2023-11-01T04:16:23.431 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.853045+0000", 2023-11-01T04:16:23.431 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 1.3360642149999999, 2023-11-01T04:16:23.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30061494799999999, 2023-11-01T04:16:23.431 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:23.431 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:23.432 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:23.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:23.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853045+0000", 2023-11-01T04:16:23.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:23.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853045+0000", 2023-11-01T04:16:23.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:23.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:23.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853046+0000", 2023-11-01T04:16:23.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8700000000000004e-07 2023-11-01T04:16:23.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:23.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853049+0000", 2023-11-01T04:16:23.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.54e-06 2023-11-01T04:16:23.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:23.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853058+0000", 2023-11-01T04:16:23.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1540000000000008e-06 2023-11-01T04:16:23.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:23.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853064+0000", 2023-11-01T04:16:23.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6459999999999998e-06 2023-11-01T04:16:23.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:23.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:22.153623+0000", 2023-11-01T04:16:23.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30055869000000002 2023-11-01T04:16:23.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:23.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:23.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:23.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:22.153660+0000", 2023-11-01T04:16:23.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7030999999999998e-05 2023-11-01T04:16:23.440 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.440 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.440 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.441 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:23.441 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:23.441 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:23.442 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:23.443 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-01T04:16:23.443 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:23.444 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-01T04:16:23.444 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-01T04:16:23.445 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:23.445 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-01T04:16:23.445 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:23.446 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-01T04:16:23.446 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:23.447 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-01T04:16:23.447 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:23.447 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-01T04:16:23.448 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:23.448 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-01T04:16:23.449 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:23.449 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-01T04:16:23.450 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:23.450 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-01T04:16:23.450 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:23.437+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:23.481 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:23.482 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-01T04:16:23.543 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:23.541+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:23.610 INFO:teuthology.orchestra.run.smithi153.stdout:osd.12: {} 2023-11-01T04:16:23.611 INFO:teuthology.orchestra.run.smithi153.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-01T04:16:23.622 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:23.623 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:23.623 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2023-11-01T04:16:23.623 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:23.639 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-01T04:16:23.648 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:23.648 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-01T04:16:23.683 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:23.684 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-01T04:16:23.687 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:23.687+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:23.708 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:23.709 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-01T04:16:23.745 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:23.744+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:23.761 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:23.761 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:23.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-01T04:16:23.863 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:23.863+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:23.944 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:23.944 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:23.948 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:23.947+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:23.958 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-01T04:16:24.008 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.4 to 1 (10000) 2023-11-01T04:16:24.018 INFO:tasks.rados.rados.0.smithi190.stdout:107: done (3 left) 2023-11-01T04:16:24.019 INFO:tasks.rados.rados.0.smithi190.stdout:108: rmattr oid 30 current snap is 5 2023-11-01T04:16:24.019 INFO:tasks.rados.rados.0.smithi190.stdout:108: done (3 left) 2023-11-01T04:16:24.019 INFO:tasks.rados.rados.0.smithi190.stdout:109: read oid 42 snap 2 2023-11-01T04:16:24.020 INFO:tasks.rados.rados.0.smithi190.stdout:109: expect (ObjNum 41 snap 0 seq_num 41) 2023-11-01T04:16:24.020 INFO:tasks.rados.rados.0.smithi190.stdout:110: rmattr oid 14 current snap is 5 2023-11-01T04:16:24.020 INFO:tasks.rados.rados.0.smithi190.stdout:110: done (4 left) 2023-11-01T04:16:24.020 INFO:tasks.rados.rados.0.smithi190.stdout:111: read oid 9 snap 2 2023-11-01T04:16:24.021 INFO:tasks.rados.rados.0.smithi190.stdout:111: expect (ObjNum 8 snap 0 seq_num 8) 2023-11-01T04:16:24.021 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 24 current snap is 5 2023-11-01T04:16:24.021 INFO:tasks.rados.rados.0.smithi190.stdout:112: seq_num 59 ranges {1245184=622592,1867776=376832} 2023-11-01T04:16:24.032 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2023-11-01T04:16:24.049 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:24.047+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:24.049 INFO:tasks.rados.rados.0.smithi190.stdout:112: writing smithi19039307-24 from 1245184 to 1867776 tid 1 2023-11-01T04:16:24.055 INFO:tasks.rados.rados.0.smithi190.stdout:112: writing smithi19039307-24 from 1867776 to 2244608 tid 2 2023-11-01T04:16:24.056 INFO:tasks.rados.rados.0.smithi190.stdout:113: copy_from oid 31 from oid 6 current snap is 5 2023-11-01T04:16:24.056 INFO:tasks.rados.rados.0.smithi190.stdout:104: expect (ObjNum 45 snap 0 seq_num 45) 2023-11-01T04:16:24.094 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:24.095 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:24.110 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-01T04:16:24.111 INFO:tasks.rados.rados.0.smithi190.stdout:105: finishing copy_from racing read to smithi19039307-36 2023-11-01T04:16:24.111 INFO:tasks.rados.rados.0.smithi190.stdout:105: finishing copy_from to smithi19039307-36 2023-11-01T04:16:24.111 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 36 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2023-11-01T04:16:24.111 INFO:tasks.rados.rados.0.smithi190.stdout:103: done (6 left) 2023-11-01T04:16:24.112 INFO:tasks.rados.rados.0.smithi190.stdout:104: done (5 left) 2023-11-01T04:16:24.112 INFO:tasks.rados.rados.0.smithi190.stdout:105: done (4 left) 2023-11-01T04:16:24.112 INFO:tasks.rados.rados.0.smithi190.stdout:114: snap_remove snap 4 2023-11-01T04:16:24.150 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:24.150+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:24.227 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:24.227 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:24.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-01T04:16:24.287 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:24.285+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:24.351 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:24.347+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:24.358 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:24.358 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:24.373 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-01T04:16:24.452 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:24.451+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:24.541 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:24.541 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:24.553 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:24.549+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:24.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-01T04:16:24.654 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:24.654+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:24.721 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:24.721 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:24.735 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-01T04:16:24.743 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.5 to 1 (10000) 2023-11-01T04:16:24.751 INFO:tasks.rados.rados.0.smithi190.stdout:114: done (4 left) 2023-11-01T04:16:24.751 INFO:tasks.rados.rados.0.smithi190.stdout:115: copy_from oid 25 from oid 30 current snap is 5 2023-11-01T04:16:24.751 INFO:tasks.rados.rados.0.smithi190.stdout:111: expect (ObjNum 8 snap 0 seq_num 8) 2023-11-01T04:16:24.754 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:16:24.751+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:16:24.763 INFO:tasks.rados.rados.0.smithi190.stdout:112: finishing write tid 1 to smithi19039307-24 2023-11-01T04:16:24.763 INFO:tasks.rados.rados.0.smithi190.stdout:113: finishing copy_from racing read to smithi19039307-31 2023-11-01T04:16:24.763 INFO:tasks.rados.rados.0.smithi190.stdout:111: done (4 left) 2023-11-01T04:16:24.764 INFO:tasks.rados.rados.0.smithi190.stdout:116: read oid 10 snap 2 2023-11-01T04:16:24.764 INFO:tasks.rados.rados.0.smithi190.stdout:116: expect (ObjNum 9 snap 0 seq_num 9) 2023-11-01T04:16:24.764 INFO:tasks.rados.rados.0.smithi190.stdout:109: expect (ObjNum 41 snap 0 seq_num 41) 2023-11-01T04:16:24.769 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2023-11-01T04:16:24.812 INFO:tasks.rados.rados.0.smithi190.stdout:112: finishing write tid 2 to smithi19039307-24 2023-11-01T04:16:24.812 INFO:tasks.rados.rados.0.smithi190.stdout:112: finishing write tid 3 to smithi19039307-24 2023-11-01T04:16:24.813 INFO:tasks.rados.rados.0.smithi190.stdout:113: finishing copy_from to smithi19039307-31 2023-11-01T04:16:24.813 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 31 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2023-11-01T04:16:24.813 INFO:tasks.rados.rados.0.smithi190.stdout:112: finishing write tid 4 to smithi19039307-24 2023-11-01T04:16:24.813 INFO:tasks.rados.rados.0.smithi190.stdout:112: oid 24 updating version 0 to 29 2023-11-01T04:16:24.813 INFO:tasks.rados.rados.0.smithi190.stdout:112: oid 24 version 29 is already newer than 27 2023-11-01T04:16:24.813 INFO:tasks.rados.rados.0.smithi190.stdout:112: oid 24 version 29 is already newer than 28 2023-11-01T04:16:24.814 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 24 v 29 (ObjNum 59 snap 5 seq_num 59) dirty exists 2023-11-01T04:16:24.814 INFO:tasks.rados.rados.0.smithi190.stdout:112: left oid 24 (ObjNum 59 snap 5 seq_num 59) 2023-11-01T04:16:24.814 INFO:tasks.rados.rados.0.smithi190.stdout:109: done (4 left) 2023-11-01T04:16:24.814 INFO:tasks.rados.rados.0.smithi190.stdout:112: done (3 left) 2023-11-01T04:16:24.814 INFO:tasks.rados.rados.0.smithi190.stdout:113: done (2 left) 2023-11-01T04:16:24.814 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 49 current snap is 5 2023-11-01T04:16:24.815 INFO:tasks.rados.rados.0.smithi190.stdout:117: seq_num 60 ranges {98304=638976,737280=376832} 2023-11-01T04:16:24.824 INFO:tasks.rados.rados.0.smithi190.stdout:117: writing smithi19039307-49 from 98304 to 737280 tid 1 2023-11-01T04:16:24.830 INFO:tasks.rados.rados.0.smithi190.stdout:117: writing smithi19039307-49 from 737280 to 1114112 tid 2 2023-11-01T04:16:24.830 INFO:tasks.rados.rados.0.smithi190.stdout:118: delete oid 43 current snap is 5 2023-11-01T04:16:24.855 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:24.851+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:24.860 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:24.860 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:24.876 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-01T04:16:24.955 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:24.953+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:24.959 INFO:tasks.rados.rados.0.smithi190.stdout:118: done (3 left) 2023-11-01T04:16:24.959 INFO:tasks.rados.rados.0.smithi190.stdout:119: read oid 31 snap -1 2023-11-01T04:16:24.959 INFO:tasks.rados.rados.0.smithi190.stdout:119: expect (ObjNum 5 snap 0 seq_num 5) 2023-11-01T04:16:24.959 INFO:tasks.rados.rados.0.smithi190.stdout:120: rollback oid 34 current snap is 5 2023-11-01T04:16:24.959 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 34 to 2 2023-11-01T04:16:24.960 INFO:tasks.rados.rados.0.smithi190.stdout:121: delete oid 17 current snap is 5 2023-11-01T04:16:24.968 INFO:tasks.rados.rados.0.smithi190.stdout:116: expect (ObjNum 9 snap 0 seq_num 9) 2023-11-01T04:16:24.998 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:24.999 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:25.014 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-01T04:16:25.027 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-01T04:16:25.056 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:16:25.055+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:16:25.157 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:25.155+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:25.170 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:25.170 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:25.180 INFO:tasks.rados.rados.0.smithi190.stdout:115: finishing copy_from racing read to smithi19039307-25 2023-11-01T04:16:25.185 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-01T04:16:25.247 INFO:tasks.rados.rados.0.smithi190.stdout:117: finishing write tid 1 to smithi19039307-49 2023-11-01T04:16:25.248 INFO:tasks.rados.rados.0.smithi190.stdout:117: finishing write tid 2 to smithi19039307-49 2023-11-01T04:16:25.257 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:25.255+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:25.264 INFO:tasks.rados.rados.0.smithi190.stdout:120: finishing rollback tid 0 to smithi19039307-34 2023-11-01T04:16:25.264 INFO:tasks.rados.rados.0.smithi190.stdout:120: finishing rollback tid 1 to smithi19039307-34 2023-11-01T04:16:25.265 INFO:tasks.rados.rados.0.smithi190.stdout:120: finishing rollback tid 2 to smithi19039307-34 2023-11-01T04:16:25.265 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 34 v 10 (ObjNum 33 snap 0 seq_num 33) dirty exists 2023-11-01T04:16:25.267 INFO:tasks.rados.rados.0.smithi190.stdout:116: done (5 left) 2023-11-01T04:16:25.267 INFO:tasks.rados.rados.0.smithi190.stdout:120: done (4 left) 2023-11-01T04:16:25.267 INFO:tasks.rados.rados.0.smithi190.stdout:121: done (3 left) 2023-11-01T04:16:25.267 INFO:tasks.rados.rados.0.smithi190.stdout:122: read oid 19 snap 2 2023-11-01T04:16:25.268 INFO:tasks.rados.rados.0.smithi190.stdout:122: expect (ObjNum 18 snap 0 seq_num 18) 2023-11-01T04:16:25.268 INFO:tasks.rados.rados.0.smithi190.stdout:123: rollback oid 17 current snap is 5 2023-11-01T04:16:25.268 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 17 to 2 2023-11-01T04:16:25.268 INFO:tasks.rados.rados.0.smithi190.stdout:124: delete oid 18 current snap is 5 2023-11-01T04:16:25.298 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:25.298 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:25.312 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-01T04:16:25.359 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:25.357+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:25.370 INFO:tasks.rados.rados.0.smithi190.stdout:117: finishing write tid 3 to smithi19039307-49 2023-11-01T04:16:25.372 INFO:tasks.rados.rados.0.smithi190.stdout:119: expect (ObjNum 5 snap 0 seq_num 5) 2023-11-01T04:16:25.380 INFO:tasks.rados.rados.0.smithi190.stdout:119: done (5 left) 2023-11-01T04:16:25.380 INFO:tasks.rados.rados.0.smithi190.stdout:124: done (4 left) 2023-11-01T04:16:25.380 INFO:tasks.rados.rados.0.smithi190.stdout:125: setattr oid 12 current snap is 5 2023-11-01T04:16:25.385 INFO:tasks.rados.rados.0.smithi190.stdout:126: snap_create 2023-11-01T04:16:25.426 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:25.426 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:25.439 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-01T04:16:25.459 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:25.455+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:25.469 INFO:tasks.rados.rados.0.smithi190.stdout:115: finishing copy_from to smithi19039307-25 2023-11-01T04:16:25.469 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 25 v 4 (ObjNum 29 snap 0 seq_num 29) dirty exists 2023-11-01T04:16:25.471 INFO:tasks.rados.rados.0.smithi190.stdout:122: expect (ObjNum 18 snap 0 seq_num 18) 2023-11-01T04:16:25.495 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 12 v 16 (ObjNum 11 snap 0 seq_num 11) dirty exists 2023-11-01T04:16:25.560 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:25.559+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:25.602 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:25.603 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:25.616 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-01T04:16:25.663 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:25.661+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:25.675 INFO:tasks.rados.rados.0.smithi190.stdout:123: finishing rollback tid 1 to smithi19039307-17 2023-11-01T04:16:25.675 INFO:tasks.rados.rados.0.smithi190.stdout:123: finishing rollback tid 2 to smithi19039307-17 2023-11-01T04:16:25.676 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 17 v 15 (ObjNum 16 snap 0 seq_num 16) dirty exists 2023-11-01T04:16:25.733 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:25.733 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:25.735 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-01T04:16:25.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-01T04:16:25.751 INFO:tasks.rados.rados.0.smithi190.stdout:117: finishing write tid 4 to smithi19039307-49 2023-11-01T04:16:25.751 INFO:tasks.rados.rados.0.smithi190.stdout:117: oid 49 updating version 0 to 38 2023-11-01T04:16:25.752 INFO:tasks.rados.rados.0.smithi190.stdout:117: oid 49 version 38 is already newer than 37 2023-11-01T04:16:25.752 INFO:tasks.rados.rados.0.smithi190.stdout:117: oid 49 updating version 38 to 39 2023-11-01T04:16:25.752 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 49 v 39 (ObjNum 60 snap 5 seq_num 60) dirty exists 2023-11-01T04:16:25.752 INFO:tasks.rados.rados.0.smithi190.stdout:117: left oid 49 (ObjNum 60 snap 5 seq_num 60) 2023-11-01T04:16:25.793 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:25.794+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:25.861 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:25.857+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:25.877 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:25.878 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:25.879 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:25.879 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-11-01T04:16:25.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-01T04:16:25.898 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:25.899 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-01T04:16:25.918 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:25.919 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-01T04:16:25.941 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:25.941 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-01T04:16:25.962 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:25.963 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-01T04:16:25.977 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:25.978+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:25.981 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:25.981 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-01T04:16:26.019 INFO:teuthology.orchestra.run.smithi153.stderr:noscrub is unset 2023-11-01T04:16:26.020 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.6 to 1 (10000) 2023-11-01T04:16:26.020 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:26.020 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-01T04:16:26.026 INFO:tasks.rados.rados.0.smithi190.stdout:115: done (5 left) 2023-11-01T04:16:26.026 INFO:tasks.rados.rados.0.smithi190.stdout:117: done (4 left) 2023-11-01T04:16:26.026 INFO:tasks.rados.rados.0.smithi190.stdout:122: done (3 left) 2023-11-01T04:16:26.026 INFO:tasks.rados.rados.0.smithi190.stdout:123: done (2 left) 2023-11-01T04:16:26.026 INFO:tasks.rados.rados.0.smithi190.stdout:125: done (1 left) 2023-11-01T04:16:26.027 INFO:tasks.rados.rados.0.smithi190.stdout:126: done (0 left) 2023-11-01T04:16:26.027 INFO:tasks.rados.rados.0.smithi190.stdout:127: copy_from oid 25 from oid 6 current snap is 6 2023-11-01T04:16:26.027 INFO:tasks.rados.rados.0.smithi190.stdout:128: setattr oid 23 current snap is 6 2023-11-01T04:16:26.031 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 15 current snap is 6 2023-11-01T04:16:26.031 INFO:tasks.rados.rados.0.smithi190.stdout:129: seq_num 64 ranges {245760=704512,950272=376832} 2023-11-01T04:16:26.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2023-11-01T04:16:26.045 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:26.045 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-01T04:16:26.048 INFO:tasks.rados.rados.0.smithi190.stdout:129: writing smithi19039307-15 from 245760 to 950272 tid 1 2023-11-01T04:16:26.055 INFO:tasks.rados.rados.0.smithi190.stdout:129: writing smithi19039307-15 from 950272 to 1327104 tid 2 2023-11-01T04:16:26.055 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 20 current snap is 6 2023-11-01T04:16:26.055 INFO:tasks.rados.rados.0.smithi190.stdout:130: seq_num 65 ranges {491520=720896,1212416=376832} 2023-11-01T04:16:26.057 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:26.058 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:26.058 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:16:26.058 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:16:26.058 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:26.062 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:26.062+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:26.063 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:26.063 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-01T04:16:26.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-01T04:16:26.075 INFO:tasks.rados.rados.0.smithi190.stdout:130: writing smithi19039307-20 from 491520 to 1212416 tid 1 2023-11-01T04:16:26.080 INFO:tasks.rados.rados.0.smithi190.stdout:130: writing smithi19039307-20 from 1212416 to 1589248 tid 2 2023-11-01T04:16:26.081 INFO:tasks.rados.rados.0.smithi190.stdout:131: rollback oid 31 current snap is 6 2023-11-01T04:16:26.081 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 31 to 2 2023-11-01T04:16:26.081 INFO:tasks.rados.rados.0.smithi190.stdout:132: read oid 24 snap -1 2023-11-01T04:16:26.081 INFO:tasks.rados.rados.0.smithi190.stdout:132: expect (ObjNum 59 snap 5 seq_num 59) 2023-11-01T04:16:26.081 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 39 current snap is 6 2023-11-01T04:16:26.082 INFO:tasks.rados.rados.0.smithi190.stdout:133: seq_num 66 ranges {638976=737280,1376256=376832} 2023-11-01T04:16:26.082 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:26.082 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-01T04:16:26.095 INFO:tasks.rados.rados.0.smithi190.stdout:133: writing smithi19039307-39 from 638976 to 1376256 tid 1 2023-11-01T04:16:26.099 INFO:tasks.rados.rados.0.smithi190.stdout:133: writing smithi19039307-39 from 1376256 to 1753088 tid 2 2023-11-01T04:16:26.099 INFO:tasks.rados.rados.0.smithi190.stdout:134: snap_create 2023-11-01T04:16:26.102 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:26.102 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-01T04:16:26.163 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:26.159+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:26.182 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:26.183 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-01T04:16:26.200 INFO:teuthology.orchestra.run.smithi153.stdout:osd.12: {} 2023-11-01T04:16:26.201 INFO:teuthology.orchestra.run.smithi153.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-01T04:16:26.245 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:26.245 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:16:26.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:16:26.245 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:16:26.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.246 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.as2 36/31 ECSubWrite(tid=23, reqid=client.4559.0:139, at_version=36'3, trim_to=0'0, roll_forward_to=34'2))", 2023-11-01T04:16:26.246 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.196944+0000", 2023-11-01T04:16:26.246 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 11.046438061, 2023-11-01T04:16:26.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102992326, 2023-11-01T04:16:26.247 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.247 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.247 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.196944+0000", 2023-11-01T04:16:26.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.196944+0000", 2023-11-01T04:16:26.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.196944+0000", 2023-11-01T04:16:26.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3300000000000003e-07 2023-11-01T04:16:26.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.197700+0000", 2023-11-01T04:16:26.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00075622600000000003 2023-11-01T04:16:26.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.197728+0000", 2023-11-01T04:16:26.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7991999999999999e-05 2023-11-01T04:16:26.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.197744+0000", 2023-11-01T04:16:26.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5109000000000001e-05 2023-11-01T04:16:26.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.197909+0000", 2023-11-01T04:16:26.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000165217 2023-11-01T04:16:26.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.197937+0000", 2023-11-01T04:16:26.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8717999999999998e-05 2023-11-01T04:16:26.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.299866+0000", 2023-11-01T04:16:26.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10192886399999999 2023-11-01T04:16:26.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.299936+0000", 2023-11-01T04:16:26.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9766999999999998e-05 2023-11-01T04:16:26.255 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.256 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.256 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.256 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/31 ECSubWrite(tid=18, reqid=client.4559.0:125, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:26.257 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.281930+0000", 2023-11-01T04:16:26.257 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.961451726, 2023-11-01T04:16:26.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20154303100000001, 2023-11-01T04:16:26.257 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.257 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.257 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281930+0000", 2023-11-01T04:16:26.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281930+0000", 2023-11-01T04:16:26.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281931+0000", 2023-11-01T04:16:26.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6400000000000001e-07 2023-11-01T04:16:26.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281961+0000", 2023-11-01T04:16:26.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9621999999999999e-05 2023-11-01T04:16:26.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281987+0000", 2023-11-01T04:16:26.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6316e-05 2023-11-01T04:16:26.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382103+0000", 2023-11-01T04:16:26.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100116649 2023-11-01T04:16:26.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.482335+0000", 2023-11-01T04:16:26.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100231813 2023-11-01T04:16:26.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.482364+0000", 2023-11-01T04:16:26.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8829000000000001e-05 2023-11-01T04:16:26.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.483427+0000", 2023-11-01T04:16:26.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010627970000000001 2023-11-01T04:16:26.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.483473+0000", 2023-11-01T04:16:26.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6241e-05 2023-11-01T04:16:26.266 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.267 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.267 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.267 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/31 ECSubWrite(tid=4, reqid=client.4559.0:136, at_version=36'3, trim_to=0'0, roll_forward_to=33'2))", 2023-11-01T04:16:26.267 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.284706+0000", 2023-11-01T04:16:26.268 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.958675818, 2023-11-01T04:16:26.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202677317, 2023-11-01T04:16:26.268 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.268 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.268 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.284706+0000", 2023-11-01T04:16:26.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.284706+0000", 2023-11-01T04:16:26.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.284707+0000", 2023-11-01T04:16:26.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8599999999999998e-07 2023-11-01T04:16:26.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.285485+0000", 2023-11-01T04:16:26.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077885900000000002 2023-11-01T04:16:26.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.285511+0000", 2023-11-01T04:16:26.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5378999999999999e-05 2023-11-01T04:16:26.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385600+0000", 2023-11-01T04:16:26.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100089631 2023-11-01T04:16:26.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385749+0000", 2023-11-01T04:16:26.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014876399999999999 2023-11-01T04:16:26.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385771+0000", 2023-11-01T04:16:26.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1535e-05 2023-11-01T04:16:26.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.487315+0000", 2023-11-01T04:16:26.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101544735 2023-11-01T04:16:26.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.487383+0000", 2023-11-01T04:16:26.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7928000000000004e-05 2023-11-01T04:16:26.277 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.278 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.278 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.278 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 36/31 ECSubWrite(tid=14, reqid=client.4559.0:146, at_version=36'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:26.279 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.322569+0000", 2023-11-01T04:16:26.279 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.920812924, 2023-11-01T04:16:26.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20180657299999999, 2023-11-01T04:16:26.279 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.279 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.279 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322569+0000", 2023-11-01T04:16:26.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322569+0000", 2023-11-01T04:16:26.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322571+0000", 2023-11-01T04:16:26.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8029999999999999e-06 2023-11-01T04:16:26.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322759+0000", 2023-11-01T04:16:26.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018862300000000001 2023-11-01T04:16:26.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322790+0000", 2023-11-01T04:16:26.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0712e-05 2023-11-01T04:16:26.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322807+0000", 2023-11-01T04:16:26.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6566e-05 2023-11-01T04:16:26.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.423031+0000", 2023-11-01T04:16:26.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10022463600000001 2023-11-01T04:16:26.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.423046+0000", 2023-11-01T04:16:26.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4632e-05 2023-11-01T04:16:26.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524322+0000", 2023-11-01T04:16:26.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101275743 2023-11-01T04:16:26.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524376+0000", 2023-11-01T04:16:26.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3857999999999998e-05 2023-11-01T04:16:26.288 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.288 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.288 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.289 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 36/31 ECSubWrite(tid=16, reqid=client.4559.0:147, at_version=36'10, trim_to=0'0, roll_forward_to=36'8))", 2023-11-01T04:16:26.289 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.382654+0000", 2023-11-01T04:16:26.289 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.860728323, 2023-11-01T04:16:26.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14175697400000001, 2023-11-01T04:16:26.290 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.290 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.290 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382654+0000", 2023-11-01T04:16:26.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382654+0000", 2023-11-01T04:16:26.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382655+0000", 2023-11-01T04:16:26.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.933e-06 2023-11-01T04:16:26.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382823+0000", 2023-11-01T04:16:26.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000167806 2023-11-01T04:16:26.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382856+0000", 2023-11-01T04:16:26.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2255999999999997e-05 2023-11-01T04:16:26.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382926+0000", 2023-11-01T04:16:26.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0759999999999993e-05 2023-11-01T04:16:26.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.424403+0000", 2023-11-01T04:16:26.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.041477075000000002 2023-11-01T04:16:26.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.424416+0000", 2023-11-01T04:16:26.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2249000000000001e-05 2023-11-01T04:16:26.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524381+0000", 2023-11-01T04:16:26.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.099965719999999994 2023-11-01T04:16:26.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524411+0000", 2023-11-01T04:16:26.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9175000000000001e-05 2023-11-01T04:16:26.298 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.299 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.299 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.299 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 36/31 ECSubWrite(tid=17, reqid=client.4559.0:148, at_version=36'11, trim_to=0'0, roll_forward_to=36'8))", 2023-11-01T04:16:26.299 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.385040+0000", 2023-11-01T04:16:26.300 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.858341859999999, 2023-11-01T04:16:26.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14044638700000001, 2023-11-01T04:16:26.300 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.300 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.300 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385040+0000", 2023-11-01T04:16:26.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385040+0000", 2023-11-01T04:16:26.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385042+0000", 2023-11-01T04:16:26.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7129999999999999e-06 2023-11-01T04:16:26.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385087+0000", 2023-11-01T04:16:26.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4895000000000003e-05 2023-11-01T04:16:26.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385121+0000", 2023-11-01T04:16:26.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4180999999999997e-05 2023-11-01T04:16:26.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385145+0000", 2023-11-01T04:16:26.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3949000000000001e-05 2023-11-01T04:16:26.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.523994+0000", 2023-11-01T04:16:26.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13884963 2023-11-01T04:16:26.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524008+0000", 2023-11-01T04:16:26.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3719e-05 2023-11-01T04:16:26.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.525407+0000", 2023-11-01T04:16:26.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013987229999999999 2023-11-01T04:16:26.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.525486+0000", 2023-11-01T04:16:26.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9577000000000001e-05 2023-11-01T04:16:26.309 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.309 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.309 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.310 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/31 ECSubWrite(tid=5, reqid=client.4559.0:137, at_version=36'4, trim_to=0'0, roll_forward_to=33'2))", 2023-11-01T04:16:26.310 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.385648+0000", 2023-11-01T04:16:26.310 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.857734172000001, 2023-11-01T04:16:26.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102787935, 2023-11-01T04:16:26.310 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.311 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.311 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385648+0000", 2023-11-01T04:16:26.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385648+0000", 2023-11-01T04:16:26.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385649+0000", 2023-11-01T04:16:26.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1170000000000001e-06 2023-11-01T04:16:26.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385676+0000", 2023-11-01T04:16:26.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7209e-05 2023-11-01T04:16:26.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385694+0000", 2023-11-01T04:16:26.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8014999999999999e-05 2023-11-01T04:16:26.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385707+0000", 2023-11-01T04:16:26.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3332999999999999e-05 2023-11-01T04:16:26.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.486918+0000", 2023-11-01T04:16:26.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101210883 2023-11-01T04:16:26.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.486940+0000", 2023-11-01T04:16:26.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1594999999999999e-05 2023-11-01T04:16:26.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.488346+0000", 2023-11-01T04:16:26.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014057729999999999 2023-11-01T04:16:26.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.488436+0000", 2023-11-01T04:16:26.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.001e-05 2023-11-01T04:16:26.319 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.320 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.320 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.320 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.6s2 36/31 ECSubWrite(tid=22, reqid=client.4559.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=36'4))", 2023-11-01T04:16:26.320 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.688041+0000", 2023-11-01T04:16:26.321 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.555341135999999, 2023-11-01T04:16:26.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102320034, 2023-11-01T04:16:26.321 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.321 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.321 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688041+0000", 2023-11-01T04:16:26.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688041+0000", 2023-11-01T04:16:26.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688042+0000", 2023-11-01T04:16:26.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.677e-06 2023-11-01T04:16:26.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688063+0000", 2023-11-01T04:16:26.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1078999999999999e-05 2023-11-01T04:16:26.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688094+0000", 2023-11-01T04:16:26.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0839000000000001e-05 2023-11-01T04:16:26.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688111+0000", 2023-11-01T04:16:26.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6492999999999999e-05 2023-11-01T04:16:26.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.789077+0000", 2023-11-01T04:16:26.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10096656499999999 2023-11-01T04:16:26.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.789093+0000", 2023-11-01T04:16:26.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6028999999999998e-05 2023-11-01T04:16:26.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.790298+0000", 2023-11-01T04:16:26.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001204917 2023-11-01T04:16:26.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.790361+0000", 2023-11-01T04:16:26.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2434999999999994e-05 2023-11-01T04:16:26.330 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.330 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.330 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.331 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 37/31 ECSubWrite(tid=21, reqid=client.4559.0:186, at_version=37'14, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:16:26.331 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.967688+0000", 2023-11-01T04:16:26.331 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.2756933890000006, 2023-11-01T04:16:26.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102405465, 2023-11-01T04:16:26.331 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.332 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.332 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967688+0000", 2023-11-01T04:16:26.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967688+0000", 2023-11-01T04:16:26.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967690+0000", 2023-11-01T04:16:26.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.657e-06 2023-11-01T04:16:26.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967739+0000", 2023-11-01T04:16:26.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8513999999999997e-05 2023-11-01T04:16:26.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967769+0000", 2023-11-01T04:16:26.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0852999999999998e-05 2023-11-01T04:16:26.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.067914+0000", 2023-11-01T04:16:26.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100144572 2023-11-01T04:16:26.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.068061+0000", 2023-11-01T04:16:26.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014692700000000001 2023-11-01T04:16:26.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.068075+0000", 2023-11-01T04:16:26.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4279e-05 2023-11-01T04:16:26.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.069999+0000", 2023-11-01T04:16:26.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019239940000000001 2023-11-01T04:16:26.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.070094+0000", 2023-11-01T04:16:26.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4668999999999998e-05 2023-11-01T04:16:26.340 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.340 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.341 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.341 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 37/31 ECSubWrite(tid=22, reqid=client.4559.0:197, at_version=37'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:26.341 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:16:26.342 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.2031103119999997, 2023-11-01T04:16:26.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.301677274, 2023-11-01T04:16:26.342 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.342 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.342 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:16:26.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:16:26.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040273+0000", 2023-11-01T04:16:26.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3370000000000001e-06 2023-11-01T04:16:26.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040307+0000", 2023-11-01T04:16:26.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3714999999999998e-05 2023-11-01T04:16:26.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040331+0000", 2023-11-01T04:16:26.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4409e-05 2023-11-01T04:16:26.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140479+0000", 2023-11-01T04:16:26.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10014764900000001 2023-11-01T04:16:26.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.240734+0000", 2023-11-01T04:16:26.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100255645 2023-11-01T04:16:26.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.240751+0000", 2023-11-01T04:16:26.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6852000000000001e-05 2023-11-01T04:16:26.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341894+0000", 2023-11-01T04:16:26.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101143302 2023-11-01T04:16:26.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341949+0000", 2023-11-01T04:16:26.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4364999999999999e-05 2023-11-01T04:16:26.351 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.351 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.351 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.352 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 37/31 ECSubWrite(tid=23, reqid=client.4559.0:198, at_version=37'10, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:26.352 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.140534+0000", 2023-11-01T04:16:26.352 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.1028474179999996, 2023-11-01T04:16:26.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202392451, 2023-11-01T04:16:26.352 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.352 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.353 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140534+0000", 2023-11-01T04:16:26.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140534+0000", 2023-11-01T04:16:26.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140536+0000", 2023-11-01T04:16:26.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.22e-06 2023-11-01T04:16:26.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140578+0000", 2023-11-01T04:16:26.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2715e-05 2023-11-01T04:16:26.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140603+0000", 2023-11-01T04:16:26.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4609000000000001e-05 2023-11-01T04:16:26.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140616+0000", 2023-11-01T04:16:26.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2763e-05 2023-11-01T04:16:26.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341493+0000", 2023-11-01T04:16:26.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.200877214 2023-11-01T04:16:26.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341508+0000", 2023-11-01T04:16:26.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5013e-05 2023-11-01T04:16:26.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.342869+0000", 2023-11-01T04:16:26.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001361183 2023-11-01T04:16:26.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.342927+0000", 2023-11-01T04:16:26.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7734e-05 2023-11-01T04:16:26.361 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.361 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.362 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.362 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 37/31 ECSubWrite(tid=13, reqid=client.4559.0:181, at_version=37'8, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:26.362 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.248764+0000", 2023-11-01T04:16:26.362 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 8.9946180330000001, 2023-11-01T04:16:26.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20321630399999999, 2023-11-01T04:16:26.363 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.363 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.363 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.248764+0000", 2023-11-01T04:16:26.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.248764+0000", 2023-11-01T04:16:26.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.248766+0000", 2023-11-01T04:16:26.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8300000000000001e-06 2023-11-01T04:16:26.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.366 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.366 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.249255+0000", 2023-11-01T04:16:26.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00048892399999999996 2023-11-01T04:16:26.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.249286+0000", 2023-11-01T04:16:26.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1004999999999999e-05 2023-11-01T04:16:26.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.249302+0000", 2023-11-01T04:16:26.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6169999999999999e-05 2023-11-01T04:16:26.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.349516+0000", 2023-11-01T04:16:26.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10021408599999999 2023-11-01T04:16:26.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.349530+0000", 2023-11-01T04:16:26.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4472e-05 2023-11-01T04:16:26.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451931+0000", 2023-11-01T04:16:26.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10240034100000001 2023-11-01T04:16:26.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451980+0000", 2023-11-01T04:16:26.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9475999999999999e-05 2023-11-01T04:16:26.372 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.372 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.372 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.373 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 37/31 ECSubWrite(tid=14, reqid=client.4559.0:182, at_version=37'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:26.373 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.253221+0000", 2023-11-01T04:16:26.373 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 8.9901608769999992, 2023-11-01T04:16:26.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19911255999999999, 2023-11-01T04:16:26.373 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.374 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.374 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253221+0000", 2023-11-01T04:16:26.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253221+0000", 2023-11-01T04:16:26.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.376 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253222+0000", 2023-11-01T04:16:26.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1570000000000001e-06 2023-11-01T04:16:26.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253418+0000", 2023-11-01T04:16:26.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019597500000000001 2023-11-01T04:16:26.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253443+0000", 2023-11-01T04:16:26.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5055999999999999e-05 2023-11-01T04:16:26.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253459+0000", 2023-11-01T04:16:26.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5495999999999999e-05 2023-11-01T04:16:26.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451029+0000", 2023-11-01T04:16:26.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19757031799999999 2023-11-01T04:16:26.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451039+0000", 2023-11-01T04:16:26.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0062e-05 2023-11-01T04:16:26.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.452298+0000", 2023-11-01T04:16:26.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012584849999999999 2023-11-01T04:16:26.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.382 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.452334+0000", 2023-11-01T04:16:26.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6010999999999998e-05 2023-11-01T04:16:26.382 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.382 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.383 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.383 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.2s5 37/31 ECSubWrite(tid=206, reqid=client.4559.0:176, at_version=37'19, trim_to=0'0, roll_forward_to=36'16))", 2023-11-01T04:16:26.383 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.253930+0000", 2023-11-01T04:16:26.383 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 8.9894517609999998, 2023-11-01T04:16:26.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10315031600000001, 2023-11-01T04:16:26.384 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.384 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.384 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253930+0000", 2023-11-01T04:16:26.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253930+0000", 2023-11-01T04:16:26.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253930+0000", 2023-11-01T04:16:26.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5999999999999999e-07 2023-11-01T04:16:26.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.387 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.387 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.387 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.254197+0000", 2023-11-01T04:16:26.387 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026679500000000001 2023-11-01T04:16:26.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.388 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.388 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.254222+0000", 2023-11-01T04:16:26.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4453000000000001e-05 2023-11-01T04:16:26.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.389 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.389 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.254237+0000", 2023-11-01T04:16:26.389 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4946000000000001e-05 2023-11-01T04:16:26.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.389 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.389 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.390 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.354451+0000", 2023-11-01T04:16:26.390 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10021408599999999 2023-11-01T04:16:26.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.390 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.390 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.390 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.354465+0000", 2023-11-01T04:16:26.391 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4549000000000001e-05 2023-11-01T04:16:26.391 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.391 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.391 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.391 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.357011+0000", 2023-11-01T04:16:26.391 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025457520000000001 2023-11-01T04:16:26.392 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.392 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.392 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.392 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.357080+0000", 2023-11-01T04:16:26.392 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9375000000000006e-05 2023-11-01T04:16:26.392 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.393 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.393 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.393 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.393 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 37/31 ECSubWrite(tid=15, reqid=client.4559.0:183, at_version=37'10, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:26.393 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.255859+0000", 2023-11-01T04:16:26.393 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 8.9875231850000006, 2023-11-01T04:16:26.394 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19868693900000001, 2023-11-01T04:16:26.394 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.394 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.394 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.394 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.395 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.395 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255859+0000", 2023-11-01T04:16:26.395 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.395 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.395 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.396 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255859+0000", 2023-11-01T04:16:26.396 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.396 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.396 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.396 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.396 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255860+0000", 2023-11-01T04:16:26.397 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2300000000000001e-06 2023-11-01T04:16:26.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.397 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.397 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255894+0000", 2023-11-01T04:16:26.397 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3796e-05 2023-11-01T04:16:26.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255923+0000", 2023-11-01T04:16:26.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9352e-05 2023-11-01T04:16:26.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255936+0000", 2023-11-01T04:16:26.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3232e-05 2023-11-01T04:16:26.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451650+0000", 2023-11-01T04:16:26.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19571368 2023-11-01T04:16:26.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451665+0000", 2023-11-01T04:16:26.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5119e-05 2023-11-01T04:16:26.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.454503+0000", 2023-11-01T04:16:26.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028377400000000001 2023-11-01T04:16:26.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.402 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.402 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.454546+0000", 2023-11-01T04:16:26.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2790000000000002e-05 2023-11-01T04:16:26.403 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.403 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.403 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.403 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.403 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.403 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpRead(3.es2 41/31 ECSubRead(tid=31, to_read={3:706dddd5:::smithi19039307-14:head=0,159744,0}, subchunks={3:706dddd5:::smithi19039307-14:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:26.404 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:21.022133+0000", 2023-11-01T04:16:26.404 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.2212491759999997, 2023-11-01T04:16:26.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16269023099999999, 2023-11-01T04:16:26.404 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.404 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.404 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.405 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.405 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022133+0000", 2023-11-01T04:16:26.405 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.406 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.406 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022133+0000", 2023-11-01T04:16:26.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.406 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.406 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.407 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022133+0000", 2023-11-01T04:16:26.407 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2e-07 2023-11-01T04:16:26.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.407 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.407 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022144+0000", 2023-11-01T04:16:26.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.118e-05 2023-11-01T04:16:26.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.408 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022175+0000", 2023-11-01T04:16:26.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0518999999999997e-05 2023-11-01T04:16:26.409 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.409 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.409 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.409 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022204+0000", 2023-11-01T04:16:26.409 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8878999999999999e-05 2023-11-01T04:16:26.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.410 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.410 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.184034+0000", 2023-11-01T04:16:26.410 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.161829903 2023-11-01T04:16:26.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.411 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.411 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.411 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.184823+0000", 2023-11-01T04:16:26.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00078932999999999998 2023-11-01T04:16:26.411 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.411 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.412 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.412 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.412 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.412 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpRead(3.es2 41/31 ECSubRead(tid=32, to_read={3:706dddd5:::smithi19039307-14:head=0,159744,0}, subchunks={3:706dddd5:::smithi19039307-14:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:26.412 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:21.023180+0000", 2023-11-01T04:16:26.412 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.2202015089999998, 2023-11-01T04:16:26.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16201669099999999, 2023-11-01T04:16:26.413 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.413 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.413 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.414 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.414 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023180+0000", 2023-11-01T04:16:26.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.414 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.415 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023180+0000", 2023-11-01T04:16:26.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.415 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.415 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.415 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.415 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023182+0000", 2023-11-01T04:16:26.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0470000000000001e-06 2023-11-01T04:16:26.416 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.416 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.416 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023203+0000", 2023-11-01T04:16:26.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0845000000000001e-05 2023-11-01T04:16:26.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023244+0000", 2023-11-01T04:16:26.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1118999999999997e-05 2023-11-01T04:16:26.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023280+0000", 2023-11-01T04:16:26.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5812000000000002e-05 2023-11-01T04:16:26.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.184966+0000", 2023-11-01T04:16:26.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16168543899999999 2023-11-01T04:16:26.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.185197+0000", 2023-11-01T04:16:26.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023142900000000001 2023-11-01T04:16:26.420 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.420 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.420 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.421 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 41/31 ECSubWrite(tid=37, reqid=osd.10.0:36, at_version=0'0, trim_to=0'0, roll_forward_to=41'16))", 2023-11-01T04:16:26.421 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:21.338735+0000", 2023-11-01T04:16:26.421 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 4.9046472559999996, 2023-11-01T04:16:26.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102459491, 2023-11-01T04:16:26.422 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.422 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.422 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.338735+0000", 2023-11-01T04:16:26.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.423 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.423 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.423 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.423 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.338735+0000", 2023-11-01T04:16:26.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.338736+0000", 2023-11-01T04:16:26.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.567e-06 2023-11-01T04:16:26.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.338757+0000", 2023-11-01T04:16:26.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0832e-05 2023-11-01T04:16:26.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.338780+0000", 2023-11-01T04:16:26.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2719999999999999e-05 2023-11-01T04:16:26.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.427 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.427 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.338795+0000", 2023-11-01T04:16:26.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4851999999999999e-05 2023-11-01T04:16:26.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.439033+0000", 2023-11-01T04:16:26.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100238039 2023-11-01T04:16:26.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.439060+0000", 2023-11-01T04:16:26.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7729000000000002e-05 2023-11-01T04:16:26.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.441101+0000", 2023-11-01T04:16:26.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020402559999999998 2023-11-01T04:16:26.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.431 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.441194+0000", 2023-11-01T04:16:26.431 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3496000000000001e-05 2023-11-01T04:16:26.431 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.431 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.431 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.431 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.432 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 41/31 ECSubWrite(tid=37, reqid=osd.11.0:36, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2023-11-01T04:16:26.432 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:21.644001+0000", 2023-11-01T04:16:26.432 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 4.5993810230000003, 2023-11-01T04:16:26.432 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.103507961, 2023-11-01T04:16:26.432 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.433 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.433 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.644001+0000", 2023-11-01T04:16:26.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.644001+0000", 2023-11-01T04:16:26.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.435 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.644003+0000", 2023-11-01T04:16:26.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7969999999999999e-06 2023-11-01T04:16:26.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.436 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.436 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.436 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.644025+0000", 2023-11-01T04:16:26.436 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2566e-05 2023-11-01T04:16:26.436 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.437 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.437 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.437 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.644055+0000", 2023-11-01T04:16:26.437 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0199e-05 2023-11-01T04:16:26.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.437 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.438 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.438 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.644071+0000", 2023-11-01T04:16:26.438 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5925000000000001e-05 2023-11-01T04:16:26.438 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.438 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.438 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.439 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.644187+0000", 2023-11-01T04:16:26.439 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115736 2023-11-01T04:16:26.439 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.439 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.439 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.440 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.644201+0000", 2023-11-01T04:16:26.440 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4387e-05 2023-11-01T04:16:26.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.440 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.440 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.440 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.747416+0000", 2023-11-01T04:16:26.441 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.103214847 2023-11-01T04:16:26.441 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.441 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.441 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.747509+0000", 2023-11-01T04:16:26.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2503999999999994e-05 2023-11-01T04:16:26.442 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.442 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.442 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.442 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.443 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 41/31 ECSubWrite(tid=38, reqid=osd.11.0:38, at_version=41'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-01T04:16:26.443 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:21.646050+0000", 2023-11-01T04:16:26.443 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 4.5973322269999999, 2023-11-01T04:16:26.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10239485199999999, 2023-11-01T04:16:26.443 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.443 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:26.444 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.444 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.444 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.444 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646050+0000", 2023-11-01T04:16:26.444 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.445 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.445 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.445 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646050+0000", 2023-11-01T04:16:26.445 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.445 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.445 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.446 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.446 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646051+0000", 2023-11-01T04:16:26.446 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.779e-06 2023-11-01T04:16:26.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.446 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.446 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646106+0000", 2023-11-01T04:16:26.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4545e-05 2023-11-01T04:16:26.447 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.447 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.448 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646137+0000", 2023-11-01T04:16:26.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1019000000000002e-05 2023-11-01T04:16:26.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.448 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646169+0000", 2023-11-01T04:16:26.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1668999999999998e-05 2023-11-01T04:16:26.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.746437+0000", 2023-11-01T04:16:26.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10026833 2023-11-01T04:16:26.450 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.746466+0000", 2023-11-01T04:16:26.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8694999999999999e-05 2023-11-01T04:16:26.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.748355+0000", 2023-11-01T04:16:26.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001888858 2023-11-01T04:16:26.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:26.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.748444+0000", 2023-11-01T04:16:26.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9957000000000006e-05 2023-11-01T04:16:26.453 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.453 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.453 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.453 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.453 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.454 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:26.455 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:26.455 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-01T04:16:26.456 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:26.457 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-01T04:16:26.457 INFO:tasks.rados.rados.0.smithi190.stdout:129: finishing write tid 1 to smithi19039307-15 2023-11-01T04:16:26.457 INFO:tasks.rados.rados.0.smithi190.stdout:131: finishing rollback tid 0 to smithi19039307-31 2023-11-01T04:16:26.457 INFO:tasks.rados.rados.0.smithi190.stdout:131: finishing rollback tid 1 to smithi19039307-31 2023-11-01T04:16:26.457 INFO:tasks.rados.rados.0.smithi190.stdout:131: finishing rollback tid 2 to smithi19039307-31 2023-11-01T04:16:26.458 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 31 v 20 (ObjNum 30 snap 0 seq_num 30) dirty exists 2023-11-01T04:16:26.458 INFO:tasks.rados.rados.0.smithi190.stdout:130: finishing write tid 1 to smithi19039307-20 2023-11-01T04:16:26.458 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 23 v 31 (ObjNum 11 snap 0 seq_num 11) dirty exists 2023-11-01T04:16:26.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-01T04:16:26.459 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:16:26.455+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:16:26.459 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:26.460 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-01T04:16:26.489 INFO:tasks.rados.rados.0.smithi190.stdout:129: finishing write tid 2 to smithi19039307-15 2023-11-01T04:16:26.494 INFO:tasks.rados.rados.0.smithi190.stdout:127: finishing copy_from racing read to smithi19039307-25 2023-11-01T04:16:26.556 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:26.555+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:26.597 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:26.597 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:16:26.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.597 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:16:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.394729+0000", 2023-11-01T04:16:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.20102194400000001, 2023-11-01T04:16:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20106107300000001, 2023-11-01T04:16:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394729+0000", 2023-11-01T04:16:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394729+0000", 2023-11-01T04:16:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394730+0000", 2023-11-01T04:16:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.093e-06 2023-11-01T04:16:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394738+0000", 2023-11-01T04:16:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2300000000000008e-06 2023-11-01T04:16:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394760+0000", 2023-11-01T04:16:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1679999999999999e-05 2023-11-01T04:16:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.395298+0000", 2023-11-01T04:16:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053837199999999996 2023-11-01T04:16:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:339 3.8s0 3:19d11dee:::smithi19039307-39:head [append 0~737280 in=737280b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.185406+0000", 2023-11-01T04:16:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.41034503999999999, 2023-11-01T04:16:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.410483139, 2023-11-01T04:16:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 339 2023-11-01T04:16:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.185406+0000", 2023-11-01T04:16:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.185406+0000", 2023-11-01T04:16:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.185408+0000", 2023-11-01T04:16:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9530000000000002e-06 2023-11-01T04:16:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186621+0000", 2023-11-01T04:16:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012132270000000001 2023-11-01T04:16:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186644+0000", 2023-11-01T04:16:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2878e-05 2023-11-01T04:16:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186663+0000", 2023-11-01T04:16:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9650999999999999e-05 2023-11-01T04:16:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186754+0000", 2023-11-01T04:16:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0606999999999995e-05 2023-11-01T04:16:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186902+0000", 2023-11-01T04:16:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000147448 2023-11-01T04:16:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.188122+0000", 2023-11-01T04:16:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001220732 2023-11-01T04:16:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.191051+0000", 2023-11-01T04:16:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029283669999999999 2023-11-01T04:16:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:16:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.468843+0000", 2023-11-01T04:16:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.126908094, 2023-11-01T04:16:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.127172126, 2023-11-01T04:16:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468843+0000", 2023-11-01T04:16:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468843+0000", 2023-11-01T04:16:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468845+0000", 2023-11-01T04:16:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.21e-06 2023-11-01T04:16:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468857+0000", 2023-11-01T04:16:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1678999999999999e-05 2023-11-01T04:16:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468888+0000", 2023-11-01T04:16:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1788999999999997e-05 2023-11-01T04:16:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569025+0000", 2023-11-01T04:16:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100136424 2023-11-01T04:16:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:340 3.8s0 3:19d11dee:::smithi19039307-39:head [append 0~376832 in=376832b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.186694+0000", 2023-11-01T04:16:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.40905719099999999, 2023-11-01T04:16:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40939948300000001, 2023-11-01T04:16:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 340 2023-11-01T04:16:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186694+0000", 2023-11-01T04:16:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186694+0000", 2023-11-01T04:16:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186695+0000", 2023-11-01T04:16:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3400000000000001e-06 2023-11-01T04:16:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187348+0000", 2023-11-01T04:16:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00065316300000000001 2023-11-01T04:16:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187364+0000", 2023-11-01T04:16:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5979999999999999e-05 2023-11-01T04:16:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187372+0000", 2023-11-01T04:16:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6599999999999995e-06 2023-11-01T04:16:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.189420+0000", 2023-11-01T04:16:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020482959999999998 2023-11-01T04:16:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.189500+0000", 2023-11-01T04:16:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9621000000000002e-05 2023-11-01T04:16:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.190386+0000", 2023-11-01T04:16:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00088641100000000003 2023-11-01T04:16:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.191553+0000", 2023-11-01T04:16:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001166413 2023-11-01T04:16:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=28, last_complete=46'20, committed=1, applied=1))", 2023-11-01T04:16:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.569161+0000", 2023-11-01T04:16:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.026589815999999999, 2023-11-01T04:16:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027049727999999999, 2023-11-01T04:16:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569161+0000", 2023-11-01T04:16:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569161+0000", 2023-11-01T04:16:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569162+0000", 2023-11-01T04:16:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3400000000000001e-06 2023-11-01T04:16:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569173+0000", 2023-11-01T04:16:26.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0293e-05 2023-11-01T04:16:26.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569197+0000", 2023-11-01T04:16:26.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4156e-05 2023-11-01T04:16:26.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569213+0000", 2023-11-01T04:16:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6455000000000001e-05 2023-11-01T04:16:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:341 3.8s0 3:19d11dee:::smithi19039307-39:head [setxattr _header (56) in=63b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.187391+0000", 2023-11-01T04:16:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.40835960100000002, 2023-11-01T04:16:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40892007699999999, 2023-11-01T04:16:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 341 2023-11-01T04:16:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187391+0000", 2023-11-01T04:16:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187391+0000", 2023-11-01T04:16:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187392+0000", 2023-11-01T04:16:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7100000000000011e-07 2023-11-01T04:16:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187398+0000", 2023-11-01T04:16:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3329999999999996e-06 2023-11-01T04:16:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187407+0000", 2023-11-01T04:16:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6269999999999995e-06 2023-11-01T04:16:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187419+0000", 2023-11-01T04:16:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2336e-05 2023-11-01T04:16:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.265764+0000", 2023-11-01T04:16:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.078345021000000001 2023-11-01T04:16:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.265926+0000", 2023-11-01T04:16:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016117900000000001 2023-11-01T04:16:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.266578+0000", 2023-11-01T04:16:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00065216799999999995 2023-11-01T04:16:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.267885+0000", 2023-11-01T04:16:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013072699999999999 2023-11-01T04:16:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=29, last_complete=46'21, committed=1, applied=1))", 2023-11-01T04:16:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.569263+0000", 2023-11-01T04:16:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.026487336, 2023-11-01T04:16:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027162777999999999, 2023-11-01T04:16:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569263+0000", 2023-11-01T04:16:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569263+0000", 2023-11-01T04:16:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569264+0000", 2023-11-01T04:16:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7700000000000004e-07 2023-11-01T04:16:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569269+0000", 2023-11-01T04:16:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9490000000000004e-06 2023-11-01T04:16:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569282+0000", 2023-11-01T04:16:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3169999999999999e-05 2023-11-01T04:16:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569293+0000", 2023-11-01T04:16:26.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0526000000000001e-05 2023-11-01T04:16:26.660 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.660 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.660 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:342 3.8s0 3:19d11dee:::smithi19039307-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.187433+0000", 2023-11-01T04:16:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.40831767499999999, 2023-11-01T04:16:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.409078051, 2023-11-01T04:16:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for rw locks", 2023-11-01T04:16:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 342 2023-11-01T04:16:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187433+0000", 2023-11-01T04:16:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187433+0000", 2023-11-01T04:16:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187434+0000", 2023-11-01T04:16:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.75e-07 2023-11-01T04:16:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187437+0000", 2023-11-01T04:16:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1870000000000001e-06 2023-11-01T04:16:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187446+0000", 2023-11-01T04:16:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0499999999999997e-06 2023-11-01T04:16:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.287537+0000", 2023-11-01T04:16:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10009085 2023-11-01T04:16:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.288584+0000", 2023-11-01T04:16:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001047463 2023-11-01T04:16:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2023-11-01T04:16:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.288883+0000", 2023-11-01T04:16:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029857700000000001 2023-11-01T04:16:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:16:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.569435+0000", 2023-11-01T04:16:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.026315361999999998, 2023-11-01T04:16:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027177248000000001, 2023-11-01T04:16:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "no flag points reached", 2023-11-01T04:16:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569435+0000", 2023-11-01T04:16:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569435+0000", 2023-11-01T04:16:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569436+0000", 2023-11-01T04:16:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6700000000000003e-07 2023-11-01T04:16:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569441+0000", 2023-11-01T04:16:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3059999999999997e-06 2023-11-01T04:16:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569461+0000", 2023-11-01T04:16:26.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9856000000000001e-05 2023-11-01T04:16:26.676 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:327 3.8s0 3:1514c4b8:::smithi19039307-15:head [setxattr _header (56) in=63b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.057121+0000", 2023-11-01T04:16:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.53862951400000003, 2023-11-01T04:16:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.53956167099999996, 2023-11-01T04:16:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:26.679 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:26.679 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:26.679 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:26.679 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 327 2023-11-01T04:16:26.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.679 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057121+0000", 2023-11-01T04:16:26.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057121+0000", 2023-11-01T04:16:26.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057122+0000", 2023-11-01T04:16:26.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8199999999999999e-07 2023-11-01T04:16:26.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057126+0000", 2023-11-01T04:16:26.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0439999999999997e-06 2023-11-01T04:16:26.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057134+0000", 2023-11-01T04:16:26.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.456e-06 2023-11-01T04:16:26.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057141+0000", 2023-11-01T04:16:26.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.861e-06 2023-11-01T04:16:26.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.161271+0000", 2023-11-01T04:16:26.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104129201 2023-11-01T04:16:26.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:26.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.161389+0000", 2023-11-01T04:16:26.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011806 2023-11-01T04:16:26.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:26.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.161906+0000", 2023-11-01T04:16:26.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00051756700000000003 2023-11-01T04:16:26.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:26.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162616+0000", 2023-11-01T04:16:26.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00070943799999999995 2023-11-01T04:16:26.688 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.688 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.688 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.689 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:328 3.8s0 3:1514c4b8:::smithi19039307-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:26.689 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.057152+0000", 2023-11-01T04:16:26.689 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.53859896699999998, 2023-11-01T04:16:26.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.53967907599999998, 2023-11-01T04:16:26.689 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.690 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for rw locks", 2023-11-01T04:16:26.690 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:26.690 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:26.690 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:26.690 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 328 2023-11-01T04:16:26.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.691 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057152+0000", 2023-11-01T04:16:26.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057152+0000", 2023-11-01T04:16:26.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057152+0000", 2023-11-01T04:16:26.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5599999999999995e-07 2023-11-01T04:16:26.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057155+0000", 2023-11-01T04:16:26.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5050000000000002e-06 2023-11-01T04:16:26.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057163+0000", 2023-11-01T04:16:26.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1629999999999996e-06 2023-11-01T04:16:26.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057169+0000", 2023-11-01T04:16:26.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8930000000000002e-06 2023-11-01T04:16:26.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:26.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162181+0000", 2023-11-01T04:16:26.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105011939 2023-11-01T04:16:26.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2023-11-01T04:16:26.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162296+0000", 2023-11-01T04:16:26.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115392 2023-11-01T04:16:26.697 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.698 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.698 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.698 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:16:26.698 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.390077+0000", 2023-11-01T04:16:26.699 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.20567354600000001, 2023-11-01T04:16:26.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20685277899999999, 2023-11-01T04:16:26.699 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.699 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:26.699 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390077+0000", 2023-11-01T04:16:26.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390077+0000", 2023-11-01T04:16:26.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390078+0000", 2023-11-01T04:16:26.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7300000000000002e-07 2023-11-01T04:16:26.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390080+0000", 2023-11-01T04:16:26.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.463e-06 2023-11-01T04:16:26.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390091+0000", 2023-11-01T04:16:26.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0740000000000001e-05 2023-11-01T04:16:26.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390098+0000", 2023-11-01T04:16:26.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.81e-06 2023-11-01T04:16:26.704 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.704 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.705 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.705 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=27, last_complete=46'18, committed=1, applied=1))", 2023-11-01T04:16:26.705 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.390469+0000", 2023-11-01T04:16:26.705 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.205281568, 2023-11-01T04:16:26.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20653150200000001, 2023-11-01T04:16:26.706 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.706 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:26.706 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390469+0000", 2023-11-01T04:16:26.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390469+0000", 2023-11-01T04:16:26.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390471+0000", 2023-11-01T04:16:26.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7999999999999999e-06 2023-11-01T04:16:26.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390481+0000", 2023-11-01T04:16:26.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6020000000000006e-06 2023-11-01T04:16:26.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390510+0000", 2023-11-01T04:16:26.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9603000000000001e-05 2023-11-01T04:16:26.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390526+0000", 2023-11-01T04:16:26.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5699e-05 2023-11-01T04:16:26.711 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.712 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.712 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.712 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=28, last_complete=46'20, committed=1, applied=1))", 2023-11-01T04:16:26.712 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.390586+0000", 2023-11-01T04:16:26.713 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.20516478899999999, 2023-11-01T04:16:26.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20648492099999999, 2023-11-01T04:16:26.713 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.713 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:26.713 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390586+0000", 2023-11-01T04:16:26.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390586+0000", 2023-11-01T04:16:26.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390587+0000", 2023-11-01T04:16:26.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9999999999999997e-07 2023-11-01T04:16:26.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390592+0000", 2023-11-01T04:16:26.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9400000000000001e-06 2023-11-01T04:16:26.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390607+0000", 2023-11-01T04:16:26.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5216e-05 2023-11-01T04:16:26.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390617+0000", 2023-11-01T04:16:26.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8260000000000005e-06 2023-11-01T04:16:26.718 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.718 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.719 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.719 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=29, last_complete=46'21, committed=1, applied=1))", 2023-11-01T04:16:26.719 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.391131+0000", 2023-11-01T04:16:26.720 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.20461965300000001, 2023-11-01T04:16:26.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20601565199999999, 2023-11-01T04:16:26.720 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.720 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:26.720 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391131+0000", 2023-11-01T04:16:26.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391131+0000", 2023-11-01T04:16:26.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391132+0000", 2023-11-01T04:16:26.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5000000000000001e-07 2023-11-01T04:16:26.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391138+0000", 2023-11-01T04:16:26.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1999999999999999e-06 2023-11-01T04:16:26.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391156+0000", 2023-11-01T04:16:26.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7252999999999999e-05 2023-11-01T04:16:26.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391165+0000", 2023-11-01T04:16:26.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3530000000000001e-06 2023-11-01T04:16:26.725 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.725 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.726 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.726 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:16:26.726 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:16:26.726 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.20341224499999999, 2023-11-01T04:16:26.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.204878369, 2023-11-01T04:16:26.727 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.727 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:26.727 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:16:26.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:16:26.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.729 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:16:26.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7400000000000002e-07 2023-11-01T04:16:26.730 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.730 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.730 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392345+0000", 2023-11-01T04:16:26.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8690000000000001e-06 2023-11-01T04:16:26.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.731 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392362+0000", 2023-11-01T04:16:26.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6194000000000001e-05 2023-11-01T04:16:26.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392372+0000", 2023-11-01T04:16:26.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0692e-05 2023-11-01T04:16:26.732 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.732 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.733 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.733 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=28, last_complete=46'20, committed=1, applied=1))", 2023-11-01T04:16:26.733 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.393704+0000", 2023-11-01T04:16:26.733 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.20204672800000001, 2023-11-01T04:16:26.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20359017900000001, 2023-11-01T04:16:26.734 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.734 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:26.734 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393704+0000", 2023-11-01T04:16:26.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.735 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393704+0000", 2023-11-01T04:16:26.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393705+0000", 2023-11-01T04:16:26.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3699999999999999e-07 2023-11-01T04:16:26.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393711+0000", 2023-11-01T04:16:26.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6899999999999997e-06 2023-11-01T04:16:26.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393726+0000", 2023-11-01T04:16:26.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5559000000000001e-05 2023-11-01T04:16:26.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393735+0000", 2023-11-01T04:16:26.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2830000000000001e-06 2023-11-01T04:16:26.740 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.740 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.740 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.740 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=29, last_complete=46'21, committed=1, applied=1))", 2023-11-01T04:16:26.741 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:16:26.741 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.20198785, 2023-11-01T04:16:26.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20360192899999999, 2023-11-01T04:16:26.741 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:26.741 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:26.741 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:26.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:26.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:16:26.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:26.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:16:26.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:26.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:26.744 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:16:26.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7700000000000005e-07 2023-11-01T04:16:26.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:26.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393770+0000", 2023-11-01T04:16:26.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.28e-06 2023-11-01T04:16:26.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:26.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393779+0000", 2023-11-01T04:16:26.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6760000000000002e-06 2023-11-01T04:16:26.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:26.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:26.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:26.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393786+0000", 2023-11-01T04:16:26.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8229999999999997e-06 2023-11-01T04:16:26.747 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.747 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:26.747 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.748 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:26.748 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2023-11-01T04:16:26.748 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 18 2023-11-01T04:16:26.748 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:26.749 INFO:tasks.rados.rados.0.smithi190.stdout:127: finishing copy_from to smithi19039307-25 2023-11-01T04:16:26.749 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 25 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2023-11-01T04:16:26.749 INFO:tasks.rados.rados.0.smithi190.stdout:129: finishing write tid 3 to smithi19039307-15 2023-11-01T04:16:26.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-01T04:16:26.751 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:26.747+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:26.795 INFO:tasks.rados.rados.0.smithi190.stdout:132: expect (ObjNum 59 snap 5 seq_num 59) 2023-11-01T04:16:26.850 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:26.847+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:26.878 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:26.879 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:26.879 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:16:26.879 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:16:26.879 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:26.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-01T04:16:26.933 INFO:tasks.rados.rados.0.smithi190.stdout:130: finishing write tid 2 to smithi19039307-20 2023-11-01T04:16:26.933 INFO:tasks.rados.rados.0.smithi190.stdout:130: finishing write tid 3 to smithi19039307-20 2023-11-01T04:16:26.934 INFO:tasks.rados.rados.0.smithi190.stdout:133: finishing write tid 1 to smithi19039307-39 2023-11-01T04:16:26.934 INFO:tasks.rados.rados.0.smithi190.stdout:133: finishing write tid 2 to smithi19039307-39 2023-11-01T04:16:26.934 INFO:tasks.rados.rados.0.smithi190.stdout:133: finishing write tid 3 to smithi19039307-39 2023-11-01T04:16:26.934 INFO:tasks.rados.rados.0.smithi190.stdout:129: finishing write tid 4 to smithi19039307-15 2023-11-01T04:16:26.934 INFO:tasks.rados.rados.0.smithi190.stdout:129: oid 15 updating version 0 to 17 2023-11-01T04:16:26.934 INFO:tasks.rados.rados.0.smithi190.stdout:129: oid 15 version 17 is already newer than 16 2023-11-01T04:16:26.935 INFO:tasks.rados.rados.0.smithi190.stdout:129: oid 15 updating version 17 to 18 2023-11-01T04:16:26.935 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 15 v 18 (ObjNum 64 snap 6 seq_num 64) dirty exists 2023-11-01T04:16:26.935 INFO:tasks.rados.rados.0.smithi190.stdout:129: left oid 15 (ObjNum 64 snap 6 seq_num 64) 2023-11-01T04:16:26.951 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:26.947+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:26.995 INFO:tasks.rados.rados.0.smithi190.stdout:130: finishing write tid 4 to smithi19039307-20 2023-11-01T04:16:26.996 INFO:tasks.rados.rados.0.smithi190.stdout:130: oid 20 updating version 0 to 35 2023-11-01T04:16:26.996 INFO:tasks.rados.rados.0.smithi190.stdout:130: oid 20 updating version 35 to 37 2023-11-01T04:16:26.996 INFO:tasks.rados.rados.0.smithi190.stdout:130: oid 20 version 37 is already newer than 36 2023-11-01T04:16:26.996 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 20 v 37 (ObjNum 65 snap 6 seq_num 65) dirty exists 2023-11-01T04:16:26.996 INFO:tasks.rados.rados.0.smithi190.stdout:130: left oid 20 (ObjNum 65 snap 6 seq_num 65) 2023-11-01T04:16:27.015 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:27.016 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:16:27.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:16:27.016 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:16:27.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.016 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es6 36/31 ECSubWrite(tid=17, reqid=client.4559.0:148, at_version=36'11, trim_to=0'0, roll_forward_to=36'8))", 2023-11-01T04:16:27.016 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.580047+0000", 2023-11-01T04:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 11.433523576000001, 2023-11-01T04:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20175456999999999, 2023-11-01T04:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.580047+0000", 2023-11-01T04:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.580047+0000", 2023-11-01T04:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.580048+0000", 2023-11-01T04:16:27.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7999999999999995e-07 2023-11-01T04:16:27.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.580071+0000", 2023-11-01T04:16:27.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3244999999999999e-05 2023-11-01T04:16:27.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.580083+0000", 2023-11-01T04:16:27.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1669e-05 2023-11-01T04:16:27.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.580108+0000", 2023-11-01T04:16:27.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5709000000000001e-05 2023-11-01T04:16:27.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.780392+0000", 2023-11-01T04:16:27.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.200283724 2023-11-01T04:16:27.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.780407+0000", 2023-11-01T04:16:27.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4759999999999999e-05 2023-11-01T04:16:27.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.781727+0000", 2023-11-01T04:16:27.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001320026 2023-11-01T04:16:27.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.781802+0000", 2023-11-01T04:16:27.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4757000000000003e-05 2023-11-01T04:16:27.026 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.027 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.027 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.027 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:164 3.8s0 3:1b945fb3:::smithi19039307-3:head [read 0~49152,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:27.028 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.317157+0000", 2023-11-01T04:16:27.028 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.696413518, 2023-11-01T04:16:27.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105826383, 2023-11-01T04:16:27.028 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.028 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:27.028 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:27.029 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:27.029 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:27.029 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 164 2023-11-01T04:16:27.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.029 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317157+0000", 2023-11-01T04:16:27.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317157+0000", 2023-11-01T04:16:27.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317160+0000", 2023-11-01T04:16:27.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0429999999999999e-06 2023-11-01T04:16:27.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317175+0000", 2023-11-01T04:16:27.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4363e-05 2023-11-01T04:16:27.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317210+0000", 2023-11-01T04:16:27.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5917999999999997e-05 2023-11-01T04:16:27.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317227+0000", 2023-11-01T04:16:27.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6886999999999999e-05 2023-11-01T04:16:27.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317375+0000", 2023-11-01T04:16:27.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014773100000000001 2023-11-01T04:16:27.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:27.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317547+0000", 2023-11-01T04:16:27.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017194899999999999 2023-11-01T04:16:27.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.422983+0000", 2023-11-01T04:16:27.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10543649200000001 2023-11-01T04:16:27.037 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.037 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.037 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.038 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:165 3.8s0 3:1b945fb3:::smithi19039307-3:head [read 0~49152,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:27.038 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.317271+0000", 2023-11-01T04:16:27.038 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.696299652, 2023-11-01T04:16:27.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.106111465, 2023-11-01T04:16:27.039 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.039 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:27.039 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:27.039 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:27.039 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:27.040 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 165 2023-11-01T04:16:27.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.040 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317271+0000", 2023-11-01T04:16:27.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317271+0000", 2023-11-01T04:16:27.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317272+0000", 2023-11-01T04:16:27.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.516e-06 2023-11-01T04:16:27.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317281+0000", 2023-11-01T04:16:27.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7129999999999994e-06 2023-11-01T04:16:27.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317301+0000", 2023-11-01T04:16:27.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9545999999999999e-05 2023-11-01T04:16:27.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317317+0000", 2023-11-01T04:16:27.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6322999999999999e-05 2023-11-01T04:16:27.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.318199+0000", 2023-11-01T04:16:27.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00088148100000000004 2023-11-01T04:16:27.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:27.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.318460+0000", 2023-11-01T04:16:27.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026122999999999999 2023-11-01T04:16:27.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.423382+0000", 2023-11-01T04:16:27.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104922656 2023-11-01T04:16:27.048 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.048 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.048 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.049 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:166 3.8s0 3:1b945fb3:::smithi19039307-3:head [read 0~49152,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:27.049 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.317340+0000", 2023-11-01T04:16:27.049 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.696230279, 2023-11-01T04:16:27.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.405887145, 2023-11-01T04:16:27.049 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.049 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:27.050 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:27.050 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:27.050 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:27.050 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 166 2023-11-01T04:16:27.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.051 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317340+0000", 2023-11-01T04:16:27.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317340+0000", 2023-11-01T04:16:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317341+0000", 2023-11-01T04:16:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.077e-06 2023-11-01T04:16:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317350+0000", 2023-11-01T04:16:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0329999999999997e-06 2023-11-01T04:16:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317367+0000", 2023-11-01T04:16:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6612999999999998e-05 2023-11-01T04:16:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317380+0000", 2023-11-01T04:16:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2832e-05 2023-11-01T04:16:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.421004+0000", 2023-11-01T04:16:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.103624181 2023-11-01T04:16:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.421215+0000", 2023-11-01T04:16:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00021057199999999999 2023-11-01T04:16:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.723227+0000", 2023-11-01T04:16:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30201283699999998 2023-11-01T04:16:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es6 37/31 ECSubWrite(tid=20, reqid=client.4559.0:185, at_version=37'13, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:16:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.965317+0000", 2023-11-01T04:16:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.048253897, 2023-11-01T04:16:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.108117439, 2023-11-01T04:16:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965317+0000", 2023-11-01T04:16:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965317+0000", 2023-11-01T04:16:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965317+0000", 2023-11-01T04:16:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5700000000000002e-07 2023-11-01T04:16:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965378+0000", 2023-11-01T04:16:27.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0764999999999998e-05 2023-11-01T04:16:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965463+0000", 2023-11-01T04:16:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4482999999999997e-05 2023-11-01T04:16:27.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965498+0000", 2023-11-01T04:16:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5608000000000002e-05 2023-11-01T04:16:27.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966204+0000", 2023-11-01T04:16:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00070615599999999999 2023-11-01T04:16:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966231+0000", 2023-11-01T04:16:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6995000000000002e-05 2023-11-01T04:16:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.073346+0000", 2023-11-01T04:16:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.107114315 2023-11-01T04:16:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.073434+0000", 2023-11-01T04:16:27.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8460000000000003e-05 2023-11-01T04:16:27.069 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.069 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.069 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.070 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es6 37/31 ECSubWrite(tid=21, reqid=client.4559.0:186, at_version=37'14, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:16:27.070 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.967787+0000", 2023-11-01T04:16:27.070 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.045783838, 2023-11-01T04:16:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105730843, 2023-11-01T04:16:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967787+0000", 2023-11-01T04:16:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967787+0000", 2023-11-01T04:16:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967789+0000", 2023-11-01T04:16:27.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.21e-06 2023-11-01T04:16:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967838+0000", 2023-11-01T04:16:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9291e-05 2023-11-01T04:16:27.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967871+0000", 2023-11-01T04:16:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3025999999999999e-05 2023-11-01T04:16:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967897+0000", 2023-11-01T04:16:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6149e-05 2023-11-01T04:16:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.968580+0000", 2023-11-01T04:16:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00068257599999999995 2023-11-01T04:16:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.968605+0000", 2023-11-01T04:16:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4916e-05 2023-11-01T04:16:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.073446+0000", 2023-11-01T04:16:27.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104841472 2023-11-01T04:16:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.073518+0000", 2023-11-01T04:16:27.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1203000000000003e-05 2023-11-01T04:16:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.080 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(osd.11.22:1 3.8s0 3:19d11dee:::smithi19039307-39:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.172163+0000", 2023-11-01T04:16:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.8414071720000003, 2023-11-01T04:16:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20436713200000001, 2023-11-01T04:16:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.172163+0000", 2023-11-01T04:16:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.172163+0000", 2023-11-01T04:16:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.172166+0000", 2023-11-01T04:16:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5160000000000001e-06 2023-11-01T04:16:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.172177+0000", 2023-11-01T04:16:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.145e-05 2023-11-01T04:16:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.172199+0000", 2023-11-01T04:16:27.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1416e-05 2023-11-01T04:16:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.172214+0000", 2023-11-01T04:16:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4885999999999999e-05 2023-11-01T04:16:27.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.272550+0000", 2023-11-01T04:16:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100336468 2023-11-01T04:16:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.272692+0000", 2023-11-01T04:16:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014170400000000001 2023-11-01T04:16:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.376531+0000", 2023-11-01T04:16:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.103838692 2023-11-01T04:16:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 41/31 ECSubWriteReply(tid=22, last_complete=41'12, committed=1, applied=1))", 2023-11-01T04:16:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:21.445822+0000", 2023-11-01T04:16:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.5677490499999998, 2023-11-01T04:16:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20123416, 2023-11-01T04:16:27.090 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.445822+0000", 2023-11-01T04:16:27.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.445822+0000", 2023-11-01T04:16:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.445823+0000", 2023-11-01T04:16:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5370000000000001e-06 2023-11-01T04:16:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.445832+0000", 2023-11-01T04:16:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4929999999999998e-06 2023-11-01T04:16:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.445862+0000", 2023-11-01T04:16:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0824999999999997e-05 2023-11-01T04:16:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.545944+0000", 2023-11-01T04:16:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100081897 2023-11-01T04:16:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.647023+0000", 2023-11-01T04:16:27.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10107872900000001 2023-11-01T04:16:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.647056+0000", 2023-11-01T04:16:27.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2678999999999997e-05 2023-11-01T04:16:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.098 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:325 3.8s0 3:1514c4b8:::smithi19039307-15:head [append 0~704512] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.050303+0000", 2023-11-01T04:16:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.96326714199999997, 2023-11-01T04:16:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.239471665, 2023-11-01T04:16:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.099 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 325 2023-11-01T04:16:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.100 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.050303+0000", 2023-11-01T04:16:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.050303+0000", 2023-11-01T04:16:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.050305+0000", 2023-11-01T04:16:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.711e-06 2023-11-01T04:16:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.051719+0000", 2023-11-01T04:16:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014133939999999999 2023-11-01T04:16:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.051737+0000", 2023-11-01T04:16:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8810000000000001e-05 2023-11-01T04:16:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.051751+0000", 2023-11-01T04:16:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3144e-05 2023-11-01T04:16:27.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.051870+0000", 2023-11-01T04:16:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119478 2023-11-01T04:16:27.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op must wait for map", 2023-11-01T04:16:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.051883+0000", 2023-11-01T04:16:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3033999999999999e-05 2023-11-01T04:16:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.055269+0000", 2023-11-01T04:16:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0033860520000000001 2023-11-01T04:16:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.055435+0000", 2023-11-01T04:16:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000166398 2023-11-01T04:16:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057313+0000", 2023-11-01T04:16:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001877861 2023-11-01T04:16:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162142+0000", 2023-11-01T04:16:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104828961 2023-11-01T04:16:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:27.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.289568+0000", 2023-11-01T04:16:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.127425759 2023-11-01T04:16:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.289775+0000", 2023-11-01T04:16:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020706300000000001 2023-11-01T04:16:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:326 3.8s0 3:1514c4b8:::smithi19039307-15:head [append 0~376832] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.056621+0000", 2023-11-01T04:16:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.95694937599999996, 2023-11-01T04:16:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43382757900000002, 2023-11-01T04:16:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.114 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:27.115 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:27.115 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:27.115 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:27.115 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 326 2023-11-01T04:16:27.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.115 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.056621+0000", 2023-11-01T04:16:27.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.056621+0000", 2023-11-01T04:16:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.056623+0000", 2023-11-01T04:16:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2640000000000001e-06 2023-11-01T04:16:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057077+0000", 2023-11-01T04:16:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00045417599999999998 2023-11-01T04:16:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057092+0000", 2023-11-01T04:16:27.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5138e-05 2023-11-01T04:16:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057103+0000", 2023-11-01T04:16:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1117999999999999e-05 2023-11-01T04:16:27.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.159853+0000", 2023-11-01T04:16:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10274968399999999 2023-11-01T04:16:27.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:27.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.159948+0000", 2023-11-01T04:16:27.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5097000000000001e-05 2023-11-01T04:16:27.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.160626+0000", 2023-11-01T04:16:27.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00067830199999999998 2023-11-01T04:16:27.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162313+0000", 2023-11-01T04:16:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016867799999999999 2023-11-01T04:16:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:27.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.490341+0000", 2023-11-01T04:16:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32802831100000002 2023-11-01T04:16:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.490449+0000", 2023-11-01T04:16:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010770899999999999 2023-11-01T04:16:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:327 3.8s0 3:1514c4b8:::smithi19039307-15:head [setxattr _header (56) in=63b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.057121+0000", 2023-11-01T04:16:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.95644931600000005, 2023-11-01T04:16:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.63439970300000004, 2023-11-01T04:16:27.127 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 327 2023-11-01T04:16:27.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057121+0000", 2023-11-01T04:16:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057121+0000", 2023-11-01T04:16:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057122+0000", 2023-11-01T04:16:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8199999999999999e-07 2023-11-01T04:16:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057126+0000", 2023-11-01T04:16:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0439999999999997e-06 2023-11-01T04:16:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057134+0000", 2023-11-01T04:16:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.456e-06 2023-11-01T04:16:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057141+0000", 2023-11-01T04:16:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.861e-06 2023-11-01T04:16:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.161271+0000", 2023-11-01T04:16:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104129201 2023-11-01T04:16:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.161389+0000", 2023-11-01T04:16:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011806 2023-11-01T04:16:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.161906+0000", 2023-11-01T04:16:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00051756700000000003 2023-11-01T04:16:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162616+0000", 2023-11-01T04:16:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00070943799999999995 2023-11-01T04:16:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:27.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691275+0000", 2023-11-01T04:16:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52865958499999999 2023-11-01T04:16:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691521+0000", 2023-11-01T04:16:27.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024580899999999999 2023-11-01T04:16:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.139 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=28, last_complete=46'20, committed=1, applied=1))", 2023-11-01T04:16:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.390015+0000", 2023-11-01T04:16:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.62355565000000002, 2023-11-01T04:16:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20075248900000001, 2023-11-01T04:16:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.140 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390015+0000", 2023-11-01T04:16:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390015+0000", 2023-11-01T04:16:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390015+0000", 2023-11-01T04:16:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9000000000000002e-07 2023-11-01T04:16:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390018+0000", 2023-11-01T04:16:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.644e-06 2023-11-01T04:16:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390027+0000", 2023-11-01T04:16:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7760000000000003e-06 2023-11-01T04:16:27.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390033+0000", 2023-11-01T04:16:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5339999999999996e-06 2023-11-01T04:16:27.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.590730+0000", 2023-11-01T04:16:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20069706600000001 2023-11-01T04:16:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.590767+0000", 2023-11-01T04:16:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7079000000000002e-05 2023-11-01T04:16:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.147 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=29, last_complete=46'21, committed=1, applied=1))", 2023-11-01T04:16:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.390045+0000", 2023-11-01T04:16:27.148 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.62352587699999995, 2023-11-01T04:16:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.200807654, 2023-11-01T04:16:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390045+0000", 2023-11-01T04:16:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390045+0000", 2023-11-01T04:16:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390045+0000", 2023-11-01T04:16:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2300000000000002e-07 2023-11-01T04:16:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390048+0000", 2023-11-01T04:16:27.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4600000000000002e-06 2023-11-01T04:16:27.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390056+0000", 2023-11-01T04:16:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6300000000000004e-06 2023-11-01T04:16:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390067+0000", 2023-11-01T04:16:27.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0686000000000001e-05 2023-11-01T04:16:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.590827+0000", 2023-11-01T04:16:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20076060000000001 2023-11-01T04:16:27.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.590852+0000", 2023-11-01T04:16:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4955000000000001e-05 2023-11-01T04:16:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.156 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:16:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.390077+0000", 2023-11-01T04:16:27.157 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.623493348, 2023-11-01T04:16:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30096590400000001, 2023-11-01T04:16:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390077+0000", 2023-11-01T04:16:27.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390077+0000", 2023-11-01T04:16:27.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390078+0000", 2023-11-01T04:16:27.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7300000000000002e-07 2023-11-01T04:16:27.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390080+0000", 2023-11-01T04:16:27.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.463e-06 2023-11-01T04:16:27.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390091+0000", 2023-11-01T04:16:27.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0740000000000001e-05 2023-11-01T04:16:27.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390098+0000", 2023-11-01T04:16:27.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.81e-06 2023-11-01T04:16:27.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691006+0000", 2023-11-01T04:16:27.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.300908385 2023-11-01T04:16:27.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691043+0000", 2023-11-01T04:16:27.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7233000000000001e-05 2023-11-01T04:16:27.165 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.165 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.165 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.166 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=27, last_complete=46'18, committed=1, applied=1))", 2023-11-01T04:16:27.166 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.390469+0000", 2023-11-01T04:16:27.166 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.62310136999999999, 2023-11-01T04:16:27.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30110259499999997, 2023-11-01T04:16:27.167 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.167 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.167 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390469+0000", 2023-11-01T04:16:27.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390469+0000", 2023-11-01T04:16:27.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390471+0000", 2023-11-01T04:16:27.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7999999999999999e-06 2023-11-01T04:16:27.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390481+0000", 2023-11-01T04:16:27.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6020000000000006e-06 2023-11-01T04:16:27.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390510+0000", 2023-11-01T04:16:27.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9603000000000001e-05 2023-11-01T04:16:27.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390526+0000", 2023-11-01T04:16:27.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5699e-05 2023-11-01T04:16:27.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691131+0000", 2023-11-01T04:16:27.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30060469899999998 2023-11-01T04:16:27.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691572+0000", 2023-11-01T04:16:27.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00044119199999999997 2023-11-01T04:16:27.174 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.174 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.174 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.175 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=28, last_complete=46'20, committed=1, applied=1))", 2023-11-01T04:16:27.175 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.390586+0000", 2023-11-01T04:16:27.175 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.62298459100000003, 2023-11-01T04:16:27.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30212400700000003, 2023-11-01T04:16:27.175 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.176 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.176 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390586+0000", 2023-11-01T04:16:27.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390586+0000", 2023-11-01T04:16:27.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390587+0000", 2023-11-01T04:16:27.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9999999999999997e-07 2023-11-01T04:16:27.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390592+0000", 2023-11-01T04:16:27.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9400000000000001e-06 2023-11-01T04:16:27.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390607+0000", 2023-11-01T04:16:27.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5216e-05 2023-11-01T04:16:27.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390617+0000", 2023-11-01T04:16:27.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8260000000000005e-06 2023-11-01T04:16:27.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.692681+0000", 2023-11-01T04:16:27.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30206392300000001 2023-11-01T04:16:27.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.692710+0000", 2023-11-01T04:16:27.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9502e-05 2023-11-01T04:16:27.183 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.183 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.183 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.183 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=29, last_complete=46'21, committed=1, applied=1))", 2023-11-01T04:16:27.184 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.391131+0000", 2023-11-01T04:16:27.184 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.62243945499999997, 2023-11-01T04:16:27.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30166817800000001, 2023-11-01T04:16:27.184 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.184 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.184 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391131+0000", 2023-11-01T04:16:27.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391131+0000", 2023-11-01T04:16:27.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391132+0000", 2023-11-01T04:16:27.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5000000000000001e-07 2023-11-01T04:16:27.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391138+0000", 2023-11-01T04:16:27.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1999999999999999e-06 2023-11-01T04:16:27.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391156+0000", 2023-11-01T04:16:27.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7252999999999999e-05 2023-11-01T04:16:27.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391165+0000", 2023-11-01T04:16:27.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3530000000000001e-06 2023-11-01T04:16:27.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.692767+0000", 2023-11-01T04:16:27.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30160192600000002 2023-11-01T04:16:27.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.692799+0000", 2023-11-01T04:16:27.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2496000000000002e-05 2023-11-01T04:16:27.191 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.191 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.192 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.192 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:16:27.192 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:16:27.192 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.62123204700000001, 2023-11-01T04:16:27.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40054957200000002, 2023-11-01T04:16:27.193 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.193 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.193 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:16:27.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:16:27.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:16:27.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7400000000000002e-07 2023-11-01T04:16:27.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392345+0000", 2023-11-01T04:16:27.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8690000000000001e-06 2023-11-01T04:16:27.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392362+0000", 2023-11-01T04:16:27.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6194000000000001e-05 2023-11-01T04:16:27.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392372+0000", 2023-11-01T04:16:27.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0692e-05 2023-11-01T04:16:27.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792857+0000", 2023-11-01T04:16:27.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40048482800000001 2023-11-01T04:16:27.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792888+0000", 2023-11-01T04:16:27.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1115000000000003e-05 2023-11-01T04:16:27.200 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.200 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.200 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.201 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=28, last_complete=46'20, committed=1, applied=1))", 2023-11-01T04:16:27.201 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.393704+0000", 2023-11-01T04:16:27.201 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.61986653000000003, 2023-11-01T04:16:27.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39925663099999997, 2023-11-01T04:16:27.202 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.202 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.202 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393704+0000", 2023-11-01T04:16:27.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393704+0000", 2023-11-01T04:16:27.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393705+0000", 2023-11-01T04:16:27.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3699999999999999e-07 2023-11-01T04:16:27.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393711+0000", 2023-11-01T04:16:27.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6899999999999997e-06 2023-11-01T04:16:27.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393726+0000", 2023-11-01T04:16:27.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5559000000000001e-05 2023-11-01T04:16:27.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393735+0000", 2023-11-01T04:16:27.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2830000000000001e-06 2023-11-01T04:16:27.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792931+0000", 2023-11-01T04:16:27.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39919643599999999 2023-11-01T04:16:27.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792961+0000", 2023-11-01T04:16:27.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9726e-05 2023-11-01T04:16:27.209 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.209 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.209 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.210 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=29, last_complete=46'21, committed=1, applied=1))", 2023-11-01T04:16:27.210 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:16:27.210 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.61980765199999999, 2023-11-01T04:16:27.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39925449800000001, 2023-11-01T04:16:27.211 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.211 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.211 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:16:27.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:16:27.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:16:27.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7700000000000005e-07 2023-11-01T04:16:27.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393770+0000", 2023-11-01T04:16:27.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.28e-06 2023-11-01T04:16:27.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393779+0000", 2023-11-01T04:16:27.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6760000000000002e-06 2023-11-01T04:16:27.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393786+0000", 2023-11-01T04:16:27.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8229999999999997e-06 2023-11-01T04:16:27.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792996+0000", 2023-11-01T04:16:27.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39921008299999999 2023-11-01T04:16:27.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.793017+0000", 2023-11-01T04:16:27.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1158999999999999e-05 2023-11-01T04:16:27.218 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.218 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.218 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.219 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.219 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.219 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:27.221 INFO:tasks.rados.rados.0.smithi190.stdout:127: done (7 left) 2023-11-01T04:16:27.221 INFO:tasks.rados.rados.0.smithi190.stdout:128: done (6 left) 2023-11-01T04:16:27.222 INFO:tasks.rados.rados.0.smithi190.stdout:129: done (5 left) 2023-11-01T04:16:27.222 INFO:tasks.rados.rados.0.smithi190.stdout:130: done (4 left) 2023-11-01T04:16:27.222 INFO:tasks.rados.rados.0.smithi190.stdout:131: done (3 left) 2023-11-01T04:16:27.222 INFO:tasks.rados.rados.0.smithi190.stdout:132: done (2 left) 2023-11-01T04:16:27.222 INFO:tasks.rados.rados.0.smithi190.stdout:134: done (1 left) 2023-11-01T04:16:27.222 INFO:tasks.rados.rados.0.smithi190.stdout:135: read oid 4 snap 5 2023-11-01T04:16:27.223 INFO:tasks.rados.rados.0.smithi190.stdout:135: expect (ObjNum 3 snap 0 seq_num 3) 2023-11-01T04:16:27.223 INFO:tasks.rados.rados.0.smithi190.stdout:136: read oid 34 snap 6 2023-11-01T04:16:27.223 INFO:tasks.rados.rados.0.smithi190.stdout:136: expect (ObjNum 33 snap 0 seq_num 33) 2023-11-01T04:16:27.223 INFO:tasks.rados.rados.0.smithi190.stdout:137: delete oid 14 current snap is 7 2023-11-01T04:16:27.223 INFO:tasks.rados.rados.0.smithi190.stdout:133: finishing write tid 4 to smithi19039307-39 2023-11-01T04:16:27.224 INFO:tasks.rados.rados.0.smithi190.stdout:133: oid 39 updating version 0 to 20 2023-11-01T04:16:27.224 INFO:tasks.rados.rados.0.smithi190.stdout:133: oid 39 updating version 20 to 22 2023-11-01T04:16:27.224 INFO:tasks.rados.rados.0.smithi190.stdout:133: oid 39 version 22 is already newer than 21 2023-11-01T04:16:27.224 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 39 v 22 (ObjNum 66 snap 6 seq_num 66) dirty exists 2023-11-01T04:16:27.224 INFO:tasks.rados.rados.0.smithi190.stdout:133: left oid 39 (ObjNum 66 snap 6 seq_num 66) 2023-11-01T04:16:27.224 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.7 to 1 (10000) 2023-11-01T04:16:27.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-01T04:16:27.225 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:27.218+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:27.225 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2023-11-01T04:16:27.244 INFO:tasks.rados.rados.0.smithi190.stdout:133: done (3 left) 2023-11-01T04:16:27.244 INFO:tasks.rados.rados.0.smithi190.stdout:137: done (2 left) 2023-11-01T04:16:27.245 INFO:tasks.rados.rados.0.smithi190.stdout:138: delete oid 44 current snap is 7 2023-11-01T04:16:27.321 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:27.322+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:27.395 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:27.396 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:16:27.396 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.396 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.6s6 47/31 ECSubWrite(tid=43, reqid=client.4559.0:351, at_version=47'6, trim_to=0'0, roll_forward_to=36'4))", 2023-11-01T04:16:27.396 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.248407+0000", 2023-11-01T04:16:27.396 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.14825474, 2023-11-01T04:16:27.397 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.148297183, 2023-11-01T04:16:27.397 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.397 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:27.397 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248407+0000", 2023-11-01T04:16:27.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248407+0000", 2023-11-01T04:16:27.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248408+0000", 2023-11-01T04:16:27.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.037e-06 2023-11-01T04:16:27.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248462+0000", 2023-11-01T04:16:27.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4497999999999999e-05 2023-11-01T04:16:27.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248501+0000", 2023-11-01T04:16:27.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8423000000000001e-05 2023-11-01T04:16:27.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.402 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.402 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.348639+0000", 2023-11-01T04:16:27.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100138353 2023-11-01T04:16:27.403 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.403 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.403 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.403 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.403 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2023-11-01T04:16:27.403 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 1 2023-11-01T04:16:27.404 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:27.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-01T04:16:27.442 INFO:tasks.rados.rados.0.smithi190.stdout:135: expect (ObjNum 3 snap 0 seq_num 3) 2023-11-01T04:16:27.450 INFO:tasks.rados.rados.0.smithi190.stdout:135: done (2 left) 2023-11-01T04:16:27.450 INFO:tasks.rados.rados.0.smithi190.stdout:138: done (1 left) 2023-11-01T04:16:27.451 INFO:tasks.rados.rados.0.smithi190.stdout:139: copy_from oid 27 from oid 39 current snap is 7 2023-11-01T04:16:27.451 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 31 current snap is 7 2023-11-01T04:16:27.451 INFO:tasks.rados.rados.0.smithi190.stdout:140: seq_num 69 ranges {507904=786432,1294336=376832} 2023-11-01T04:16:27.453 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:27.454+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:27.471 INFO:tasks.rados.rados.0.smithi190.stdout:140: writing smithi19039307-31 from 507904 to 1294336 tid 1 2023-11-01T04:16:27.478 INFO:tasks.rados.rados.0.smithi190.stdout:140: writing smithi19039307-31 from 1294336 to 1671168 tid 2 2023-11-01T04:16:27.478 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 50 current snap is 7 2023-11-01T04:16:27.478 INFO:tasks.rados.rados.0.smithi190.stdout:141: seq_num 70 ranges {835584=802816,1638400=376832} 2023-11-01T04:16:27.504 INFO:tasks.rados.rados.0.smithi190.stdout:141: writing smithi19039307-50 from 835584 to 1638400 tid 1 2023-11-01T04:16:27.509 INFO:tasks.rados.rados.0.smithi190.stdout:141: writing smithi19039307-50 from 1638400 to 2015232 tid 2 2023-11-01T04:16:27.509 INFO:tasks.rados.rados.0.smithi190.stdout:142: copy_from oid 19 from oid 8 current snap is 7 2023-11-01T04:16:27.510 INFO:tasks.rados.rados.0.smithi190.stdout:143: copy_from oid 37 from oid 14 current snap is 7 2023-11-01T04:16:27.510 INFO:tasks.rados.rados.0.smithi190.stdout:144: snap_create 2023-11-01T04:16:27.523 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:27.522+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:27.575 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:16:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:16:27.576 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:27.590 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-01T04:16:27.614 INFO:tasks.rados.rados.0.smithi190.stdout:143: finishing copy_from to smithi19039307-37 2023-11-01T04:16:27.614 INFO:tasks.rados.rados.0.smithi190.stdout:143: got expected ENOENT (src dne) 2023-11-01T04:16:27.615 INFO:tasks.rados.rados.0.smithi190.stdout:143: finishing copy_from racing read to smithi19039307-37 2023-11-01T04:16:27.624 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:27.620+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:27.642 INFO:tasks.rados.rados.0.smithi190.stdout:136: expect (ObjNum 33 snap 0 seq_num 33) 2023-11-01T04:16:27.688 INFO:tasks.rados.rados.0.smithi190.stdout:140: finishing write tid 1 to smithi19039307-31 2023-11-01T04:16:27.689 INFO:tasks.rados.rados.0.smithi190.stdout:140: finishing write tid 2 to smithi19039307-31 2023-11-01T04:16:27.689 INFO:tasks.rados.rados.0.smithi190.stdout:140: finishing write tid 3 to smithi19039307-31 2023-11-01T04:16:27.725 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:27.722+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:27.760 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:27.760 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:16:27.760 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:16:27.760 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:16:27.761 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.761 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.6s6 36/31 ECSubWrite(tid=20, reqid=client.4559.0:130, at_version=36'3, trim_to=0'0, roll_forward_to=35'2))", 2023-11-01T04:16:27.761 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.474824+0000", 2023-11-01T04:16:27.761 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 12.283210828, 2023-11-01T04:16:27.761 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011070396, 2023-11-01T04:16:27.762 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.762 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.762 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.762 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.762 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.762 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.474824+0000", 2023-11-01T04:16:27.763 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.763 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.763 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.763 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.763 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.474824+0000", 2023-11-01T04:16:27.764 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.764 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.764 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.764 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.764 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.474826+0000", 2023-11-01T04:16:27.765 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.013e-06 2023-11-01T04:16:27.765 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.765 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.765 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.765 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.475143+0000", 2023-11-01T04:16:27.766 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00031695399999999999 2023-11-01T04:16:27.766 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.766 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.766 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.766 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.475177+0000", 2023-11-01T04:16:27.766 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4625000000000001e-05 2023-11-01T04:16:27.767 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.767 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.767 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.767 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.475195+0000", 2023-11-01T04:16:27.767 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7635999999999999e-05 2023-11-01T04:16:27.768 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.768 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.768 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.768 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.475326+0000", 2023-11-01T04:16:27.768 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013152199999999999 2023-11-01T04:16:27.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.769 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.769 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.769 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.475341+0000", 2023-11-01T04:16:27.769 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5119e-05 2023-11-01T04:16:27.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.770 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.770 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.770 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.485833+0000", 2023-11-01T04:16:27.770 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010491864 2023-11-01T04:16:27.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.771 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.771 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.771 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.485894+0000", 2023-11-01T04:16:27.771 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0662999999999998e-05 2023-11-01T04:16:27.771 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.772 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.772 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.772 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.772 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.772 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.6s6 36/31 ECSubWrite(tid=21, reqid=client.4559.0:131, at_version=36'4, trim_to=0'0, roll_forward_to=35'2))", 2023-11-01T04:16:27.773 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.477430+0000", 2023-11-01T04:16:27.773 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 12.280604343, 2023-11-01T04:16:27.773 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0084956340000000002, 2023-11-01T04:16:27.773 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.773 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.774 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.774 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.774 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.774 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.477430+0000", 2023-11-01T04:16:27.774 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.775 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.775 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.775 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.477430+0000", 2023-11-01T04:16:27.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.776 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.776 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.776 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.477432+0000", 2023-11-01T04:16:27.776 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.716e-06 2023-11-01T04:16:27.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.777 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.777 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.777 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.477466+0000", 2023-11-01T04:16:27.777 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4588999999999997e-05 2023-11-01T04:16:27.777 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.777 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.778 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.778 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.477504+0000", 2023-11-01T04:16:27.778 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7157999999999999e-05 2023-11-01T04:16:27.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.778 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.779 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.779 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.477523+0000", 2023-11-01T04:16:27.779 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9675999999999999e-05 2023-11-01T04:16:27.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.779 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.780 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.780 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.477652+0000", 2023-11-01T04:16:27.780 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012896500000000001 2023-11-01T04:16:27.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.780 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.780 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.781 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.477667+0000", 2023-11-01T04:16:27.781 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4871999999999999e-05 2023-11-01T04:16:27.781 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.781 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.781 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.781 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.485899+0000", 2023-11-01T04:16:27.782 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0082319890000000003 2023-11-01T04:16:27.782 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.782 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.782 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.782 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.485926+0000", 2023-11-01T04:16:27.782 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6669000000000002e-05 2023-11-01T04:16:27.783 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.783 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.783 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.783 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.783 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.783 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.6s6 36/31 ECSubWrite(tid=22, reqid=client.4559.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=36'4))", 2023-11-01T04:16:27.784 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.688060+0000", 2023-11-01T04:16:27.784 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 12.06997432, 2023-11-01T04:16:27.784 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19558595100000001, 2023-11-01T04:16:27.784 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.784 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.784 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.785 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.785 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.785 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688060+0000", 2023-11-01T04:16:27.785 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.785 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.786 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.786 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.786 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688060+0000", 2023-11-01T04:16:27.786 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.786 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.786 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.787 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.787 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688062+0000", 2023-11-01T04:16:27.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1859999999999999e-06 2023-11-01T04:16:27.787 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.787 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.788 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688084+0000", 2023-11-01T04:16:27.788 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1849e-05 2023-11-01T04:16:27.788 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.789 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688115+0000", 2023-11-01T04:16:27.789 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0788999999999999e-05 2023-11-01T04:16:27.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.789 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.790 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688131+0000", 2023-11-01T04:16:27.790 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6506000000000001e-05 2023-11-01T04:16:27.790 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.790 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.790 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.790 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.788338+0000", 2023-11-01T04:16:27.791 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100206936 2023-11-01T04:16:27.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.791 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.791 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.791 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.788353+0000", 2023-11-01T04:16:27.791 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5109000000000001e-05 2023-11-01T04:16:27.792 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.792 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.883595+0000", 2023-11-01T04:16:27.792 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.095241261999999993 2023-11-01T04:16:27.793 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.793 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.793 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.793 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.883646+0000", 2023-11-01T04:16:27.793 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1313999999999997e-05 2023-11-01T04:16:27.793 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.794 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.794 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.794 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.794 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 36/31 ECSubWrite(tid=18, reqid=client.4559.0:148, at_version=0'0, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:16:27.795 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.782962+0000", 2023-11-01T04:16:27.795 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 11.975072425, 2023-11-01T04:16:27.795 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102057014, 2023-11-01T04:16:27.795 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.795 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.795 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.796 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.796 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.796 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.782962+0000", 2023-11-01T04:16:27.796 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.796 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.796 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.797 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.797 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.782962+0000", 2023-11-01T04:16:27.797 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.797 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.797 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.797 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.798 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.782964+0000", 2023-11-01T04:16:27.798 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9360000000000002e-06 2023-11-01T04:16:27.798 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.798 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.798 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.799 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.782985+0000", 2023-11-01T04:16:27.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1552999999999998e-05 2023-11-01T04:16:27.799 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.799 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.799 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.799 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.783018+0000", 2023-11-01T04:16:27.800 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2904999999999998e-05 2023-11-01T04:16:27.800 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.800 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.800 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.800 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.783044+0000", 2023-11-01T04:16:27.800 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5876000000000001e-05 2023-11-01T04:16:27.801 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.801 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.801 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.801 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.883298+0000", 2023-11-01T04:16:27.801 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100253594 2023-11-01T04:16:27.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.802 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.802 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.883313+0000", 2023-11-01T04:16:27.802 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5065000000000001e-05 2023-11-01T04:16:27.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.884941+0000", 2023-11-01T04:16:27.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001628408 2023-11-01T04:16:27.803 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.804 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.804 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.804 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.885019+0000", 2023-11-01T04:16:27.804 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7676999999999995e-05 2023-11-01T04:16:27.804 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.804 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.805 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.805 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.805 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.805 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpRead(3.es1 36/31 ECSubRead(tid=19, to_read={3:7f8a06cd:::smithi19039307-49:head=0,4096,0}, subchunks={3:7f8a06cd:::smithi19039307-49:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:27.805 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.784010+0000", 2023-11-01T04:16:27.805 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 11.974024801000001, 2023-11-01T04:16:27.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100886925, 2023-11-01T04:16:27.806 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.806 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.806 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.806 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.807 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.807 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.784010+0000", 2023-11-01T04:16:27.807 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.807 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.807 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.807 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.808 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.784010+0000", 2023-11-01T04:16:27.808 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.808 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.808 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.808 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.808 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.784012+0000", 2023-11-01T04:16:27.809 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3489999999999999e-06 2023-11-01T04:16:27.809 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.809 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.809 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.809 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.784030+0000", 2023-11-01T04:16:27.810 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8240000000000002e-05 2023-11-01T04:16:27.810 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.810 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.810 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.810 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.784071+0000", 2023-11-01T04:16:27.810 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1278e-05 2023-11-01T04:16:27.811 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.811 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.811 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.811 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.884224+0000", 2023-11-01T04:16:27.811 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100152118 2023-11-01T04:16:27.812 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.812 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.812 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.812 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.884390+0000", 2023-11-01T04:16:27.812 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016592700000000001 2023-11-01T04:16:27.812 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.813 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.884897+0000", 2023-11-01T04:16:27.813 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00050701299999999995 2023-11-01T04:16:27.813 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.813 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.814 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.814 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.814 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.814 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.as3 36/31 ECSubWrite(tid=42, reqid=client.4559.0:167, at_version=0'0, trim_to=0'0, roll_forward_to=36'5))", 2023-11-01T04:16:27.814 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.825351+0000", 2023-11-01T04:16:27.815 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.932683732999999, 2023-11-01T04:16:27.815 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101775541, 2023-11-01T04:16:27.815 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.815 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.815 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.815 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.816 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.816 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.825351+0000", 2023-11-01T04:16:27.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.816 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.816 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.817 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.825351+0000", 2023-11-01T04:16:27.817 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.818 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.825353+0000", 2023-11-01T04:16:27.818 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.95e-06 2023-11-01T04:16:27.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.818 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.818 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.825374+0000", 2023-11-01T04:16:27.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1795999999999999e-05 2023-11-01T04:16:27.819 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.820 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.825404+0000", 2023-11-01T04:16:27.820 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9726e-05 2023-11-01T04:16:27.820 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.820 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.820 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.825424+0000", 2023-11-01T04:16:27.821 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0322e-05 2023-11-01T04:16:27.821 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.821 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.821 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.821 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.925691+0000", 2023-11-01T04:16:27.821 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100266114 2023-11-01T04:16:27.822 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.822 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.822 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.822 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.925708+0000", 2023-11-01T04:16:27.822 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7169000000000001e-05 2023-11-01T04:16:27.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.823 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.927053+0000", 2023-11-01T04:16:27.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001345567 2023-11-01T04:16:27.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.824 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.824 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.824 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.927126+0000", 2023-11-01T04:16:27.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2897000000000004e-05 2023-11-01T04:16:27.824 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.824 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.825 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.825 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.825 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 37/31 ECSubWrite(tid=20, reqid=client.4559.0:185, at_version=37'13, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:16:27.825 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.965368+0000", 2023-11-01T04:16:27.825 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.792666531, 2023-11-01T04:16:27.826 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10412571, 2023-11-01T04:16:27.826 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.826 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.826 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.826 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.827 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.827 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965368+0000", 2023-11-01T04:16:27.827 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.827 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.827 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.827 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965368+0000", 2023-11-01T04:16:27.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965369+0000", 2023-11-01T04:16:27.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4600000000000004e-07 2023-11-01T04:16:27.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.829 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.829 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965431+0000", 2023-11-01T04:16:27.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2322000000000004e-05 2023-11-01T04:16:27.830 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.830 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.830 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.830 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965465+0000", 2023-11-01T04:16:27.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4408e-05 2023-11-01T04:16:27.831 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.831 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.831 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965490+0000", 2023-11-01T04:16:27.831 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4272000000000001e-05 2023-11-01T04:16:27.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.832 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.832 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.832 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965716+0000", 2023-11-01T04:16:27.832 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022658800000000001 2023-11-01T04:16:27.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.833 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.833 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965742+0000", 2023-11-01T04:16:27.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5445000000000001e-05 2023-11-01T04:16:27.833 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.069410+0000", 2023-11-01T04:16:27.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10366858299999999 2023-11-01T04:16:27.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.069494+0000", 2023-11-01T04:16:27.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3446e-05 2023-11-01T04:16:27.835 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.835 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.835 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.836 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.836 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 37/31 ECSubWrite(tid=21, reqid=client.4559.0:186, at_version=37'14, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:16:27.836 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.967795+0000", 2023-11-01T04:16:27.836 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.790239253999999, 2023-11-01T04:16:27.836 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102571579, 2023-11-01T04:16:27.836 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.837 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.837 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.837 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.837 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967795+0000", 2023-11-01T04:16:27.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967795+0000", 2023-11-01T04:16:27.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.839 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.839 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.839 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967797+0000", 2023-11-01T04:16:27.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5600000000000001e-06 2023-11-01T04:16:27.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.840 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967838+0000", 2023-11-01T04:16:27.840 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1578000000000001e-05 2023-11-01T04:16:27.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.841 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.841 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.841 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967874+0000", 2023-11-01T04:16:27.841 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5846000000000002e-05 2023-11-01T04:16:27.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.842 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.842 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.842 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967898+0000", 2023-11-01T04:16:27.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3402000000000001e-05 2023-11-01T04:16:27.842 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.842 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.843 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.843 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.068196+0000", 2023-11-01T04:16:27.843 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100298416 2023-11-01T04:16:27.843 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.843 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.843 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.844 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.068223+0000", 2023-11-01T04:16:27.844 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7248000000000001e-05 2023-11-01T04:16:27.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.844 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.844 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.844 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.070268+0000", 2023-11-01T04:16:27.845 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020447289999999999 2023-11-01T04:16:27.845 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.845 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.845 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.845 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.070367+0000", 2023-11-01T04:16:27.846 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8800000000000003e-05 2023-11-01T04:16:27.846 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.846 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.846 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.846 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.847 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 37/31 ECSubWrite(tid=22, reqid=client.4559.0:187, at_version=37'15, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:16:27.847 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.969961+0000", 2023-11-01T04:16:27.847 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.78807316, 2023-11-01T04:16:27.847 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10125532499999999, 2023-11-01T04:16:27.847 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.847 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.848 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.848 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.848 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.848 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.969961+0000", 2023-11-01T04:16:27.848 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.849 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.849 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.849 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.849 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.969961+0000", 2023-11-01T04:16:27.849 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.849 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.850 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.850 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.850 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.969963+0000", 2023-11-01T04:16:27.850 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.787e-06 2023-11-01T04:16:27.850 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.851 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.851 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.851 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970008+0000", 2023-11-01T04:16:27.851 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5064000000000001e-05 2023-11-01T04:16:27.851 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.851 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.852 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.852 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970049+0000", 2023-11-01T04:16:27.852 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0642000000000002e-05 2023-11-01T04:16:27.852 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.852 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.853 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.853 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970072+0000", 2023-11-01T04:16:27.853 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3726e-05 2023-11-01T04:16:27.853 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.853 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.853 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.854 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.069582+0000", 2023-11-01T04:16:27.854 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.099509776999999994 2023-11-01T04:16:27.854 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.854 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.854 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.854 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.069605+0000", 2023-11-01T04:16:27.855 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2476e-05 2023-11-01T04:16:27.855 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.855 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.855 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.855 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.071159+0000", 2023-11-01T04:16:27.855 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015542080000000001 2023-11-01T04:16:27.856 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.856 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.856 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.856 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.071217+0000", 2023-11-01T04:16:27.856 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7645000000000001e-05 2023-11-01T04:16:27.857 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.857 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.857 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.857 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.857 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.857 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 37/31 ECSubWrite(tid=24, reqid=client.4559.0:194, at_version=37'17, trim_to=0'0, roll_forward_to=37'15))", 2023-11-01T04:16:27.858 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.479841+0000", 2023-11-01T04:16:27.858 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.278193526000001, 2023-11-01T04:16:27.858 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102648423, 2023-11-01T04:16:27.858 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.858 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.858 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.859 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.859 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.859 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.479841+0000", 2023-11-01T04:16:27.859 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.859 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.859 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.860 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.860 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.479841+0000", 2023-11-01T04:16:27.860 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.860 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.860 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.861 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.861 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.479843+0000", 2023-11-01T04:16:27.861 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2069999999999998e-06 2023-11-01T04:16:27.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.861 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.861 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.862 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.480582+0000", 2023-11-01T04:16:27.862 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00073883 2023-11-01T04:16:27.862 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.862 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.862 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.862 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.480614+0000", 2023-11-01T04:16:27.863 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2149e-05 2023-11-01T04:16:27.863 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.863 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.863 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.863 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.480633+0000", 2023-11-01T04:16:27.863 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9315999999999999e-05 2023-11-01T04:16:27.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.864 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.864 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.864 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.580829+0000", 2023-11-01T04:16:27.864 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10019570799999999 2023-11-01T04:16:27.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.865 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.865 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.865 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.580837+0000", 2023-11-01T04:16:27.865 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9209999999999992e-06 2023-11-01T04:16:27.865 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.865 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.866 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.866 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.582452+0000", 2023-11-01T04:16:27.866 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001615099 2023-11-01T04:16:27.866 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.866 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.866 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.867 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.582489+0000", 2023-11-01T04:16:27.867 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7193000000000001e-05 2023-11-01T04:16:27.867 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.867 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.867 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.867 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.868 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.868 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 37/31 ECSubWrite(tid=25, reqid=client.4559.0:202, at_version=37'19, trim_to=0'0, roll_forward_to=37'15))", 2023-11-01T04:16:27.868 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.681455+0000", 2023-11-01T04:16:27.868 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.076579672999999, 2023-11-01T04:16:27.868 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102126551, 2023-11-01T04:16:27.868 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.869 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.869 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.869 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.869 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.869 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.681455+0000", 2023-11-01T04:16:27.869 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.870 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.870 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.870 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.870 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.681455+0000", 2023-11-01T04:16:27.870 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.871 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.871 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.871 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.681457+0000", 2023-11-01T04:16:27.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2259999999999999e-06 2023-11-01T04:16:27.871 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.872 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.681543+0000", 2023-11-01T04:16:27.872 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6299999999999997e-05 2023-11-01T04:16:27.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.872 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.873 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.681575+0000", 2023-11-01T04:16:27.873 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2125000000000002e-05 2023-11-01T04:16:27.873 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.873 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.873 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.873 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.681597+0000", 2023-11-01T04:16:27.874 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1843000000000001e-05 2023-11-01T04:16:27.874 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.874 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.874 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.874 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.781826+0000", 2023-11-01T04:16:27.874 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100228303 2023-11-01T04:16:27.875 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.875 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.875 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.875 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.781836+0000", 2023-11-01T04:16:27.875 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0419e-05 2023-11-01T04:16:27.875 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.876 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.876 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.783534+0000", 2023-11-01T04:16:27.876 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016981030000000001 2023-11-01T04:16:27.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.877 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.877 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.783581+0000", 2023-11-01T04:16:27.877 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7231999999999998e-05 2023-11-01T04:16:27.877 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.877 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.877 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.878 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.878 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.878 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 41/31 ECSubWrite(tid=36, reqid=osd.11.0:36, at_version=41'27, trim_to=0'0, roll_forward_to=39'23))", 2023-11-01T04:16:27.878 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:21.431166+0000", 2023-11-01T04:16:27.878 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 6.326868878, 2023-11-01T04:16:27.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043479800000000004, 2023-11-01T04:16:27.879 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.879 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.879 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.879 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.879 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.431166+0000", 2023-11-01T04:16:27.880 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.880 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.431166+0000", 2023-11-01T04:16:27.881 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.881 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.881 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.881 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.881 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.431167+0000", 2023-11-01T04:16:27.881 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.587e-06 2023-11-01T04:16:27.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.882 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.882 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.882 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.431229+0000", 2023-11-01T04:16:27.882 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2141e-05 2023-11-01T04:16:27.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.883 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.883 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.431265+0000", 2023-11-01T04:16:27.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5502e-05 2023-11-01T04:16:27.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.883 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.884 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.884 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.431286+0000", 2023-11-01T04:16:27.884 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0979e-05 2023-11-01T04:16:27.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.884 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.884 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.431453+0000", 2023-11-01T04:16:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016693799999999999 2023-11-01T04:16:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.431474+0000", 2023-11-01T04:16:27.885 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1722999999999999e-05 2023-11-01T04:16:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.435430+0000", 2023-11-01T04:16:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039560359999999996 2023-11-01T04:16:27.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.435514+0000", 2023-11-01T04:16:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3073999999999998e-05 2023-11-01T04:16:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.887 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.7s2 41/31 ECSubWrite(tid=36, reqid=osd.14.0:35, at_version=0'0, trim_to=0'0, roll_forward_to=41'14))", 2023-11-01T04:16:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:21.443575+0000", 2023-11-01T04:16:27.888 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 6.3144589880000002, 2023-11-01T04:16:27.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10184101600000001, 2023-11-01T04:16:27.889 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.889 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.889 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.889 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.443575+0000", 2023-11-01T04:16:27.890 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.890 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.890 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.891 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.443575+0000", 2023-11-01T04:16:27.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.891 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.891 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.891 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.443577+0000", 2023-11-01T04:16:27.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8470000000000001e-06 2023-11-01T04:16:27.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.443599+0000", 2023-11-01T04:16:27.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1305e-05 2023-11-01T04:16:27.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.443628+0000", 2023-11-01T04:16:27.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9076000000000001e-05 2023-11-01T04:16:27.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.443644+0000", 2023-11-01T04:16:27.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6708999999999999e-05 2023-11-01T04:16:27.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.895 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.443789+0000", 2023-11-01T04:16:27.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000145069 2023-11-01T04:16:27.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.443814+0000", 2023-11-01T04:16:27.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4969000000000001e-05 2023-11-01T04:16:27.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.545321+0000", 2023-11-01T04:16:27.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101507111 2023-11-01T04:16:27.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.545416+0000", 2023-11-01T04:16:27.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4930000000000001e-05 2023-11-01T04:16:27.898 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.898 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.898 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.899 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 41/31 ECSubWrite(tid=38, reqid=osd.11.0:38, at_version=41'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-01T04:16:27.899 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:21.646162+0000", 2023-11-01T04:16:27.899 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 6.1118720450000001, 2023-11-01T04:16:27.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102766129, 2023-11-01T04:16:27.899 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.899 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.900 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646162+0000", 2023-11-01T04:16:27.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646162+0000", 2023-11-01T04:16:27.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646164+0000", 2023-11-01T04:16:27.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.103e-06 2023-11-01T04:16:27.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646219+0000", 2023-11-01T04:16:27.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4981000000000001e-05 2023-11-01T04:16:27.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646272+0000", 2023-11-01T04:16:27.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2324999999999998e-05 2023-11-01T04:16:27.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646298+0000", 2023-11-01T04:16:27.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6622e-05 2023-11-01T04:16:27.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.746561+0000", 2023-11-01T04:16:27.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100262254 2023-11-01T04:16:27.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.746586+0000", 2023-11-01T04:16:27.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5015e-05 2023-11-01T04:16:27.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.748844+0000", 2023-11-01T04:16:27.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022586849999999999 2023-11-01T04:16:27.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.748928+0000", 2023-11-01T04:16:27.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4143999999999999e-05 2023-11-01T04:16:27.908 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.908 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.908 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.909 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.7s2 43/31 ECSubWrite(tid=37, reqid=osd.14.0:37, at_version=43'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-01T04:16:27.909 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:23.017682+0000", 2023-11-01T04:16:27.909 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 4.7403519249999997, 2023-11-01T04:16:27.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102610987, 2023-11-01T04:16:27.910 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.910 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.910 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.017682+0000", 2023-11-01T04:16:27.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.017682+0000", 2023-11-01T04:16:27.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.017683+0000", 2023-11-01T04:16:27.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6700000000000002e-07 2023-11-01T04:16:27.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.017731+0000", 2023-11-01T04:16:27.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7623999999999997e-05 2023-11-01T04:16:27.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.017765+0000", 2023-11-01T04:16:27.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4212999999999999e-05 2023-11-01T04:16:27.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.017790+0000", 2023-11-01T04:16:27.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4332e-05 2023-11-01T04:16:27.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.118102+0000", 2023-11-01T04:16:27.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100312568 2023-11-01T04:16:27.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.118128+0000", 2023-11-01T04:16:27.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6256000000000001e-05 2023-11-01T04:16:27.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.120214+0000", 2023-11-01T04:16:27.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002085642 2023-11-01T04:16:27.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.120293+0000", 2023-11-01T04:16:27.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9485000000000002e-05 2023-11-01T04:16:27.919 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.919 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.919 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.919 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 47/31 ECSubWrite(tid=52, reqid=client.4559.0:350, at_version=47'43, trim_to=0'0, roll_forward_to=45'41))", 2023-11-01T04:16:27.920 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.037415+0000", 2023-11-01T04:16:27.920 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.72061915799999998, 2023-11-01T04:16:27.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10750568200000001, 2023-11-01T04:16:27.920 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.920 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.920 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.037415+0000", 2023-11-01T04:16:27.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.037415+0000", 2023-11-01T04:16:27.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.037416+0000", 2023-11-01T04:16:27.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.17e-06 2023-11-01T04:16:27.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.037474+0000", 2023-11-01T04:16:27.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7584999999999998e-05 2023-11-01T04:16:27.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.037507+0000", 2023-11-01T04:16:27.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3358999999999997e-05 2023-11-01T04:16:27.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.137630+0000", 2023-11-01T04:16:27.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100122251 2023-11-01T04:16:27.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.142712+0000", 2023-11-01T04:16:27.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0050826220000000002 2023-11-01T04:16:27.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.142732+0000", 2023-11-01T04:16:27.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9619999999999998e-05 2023-11-01T04:16:27.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.144845+0000", 2023-11-01T04:16:27.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021129040000000001 2023-11-01T04:16:27.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.144921+0000", 2023-11-01T04:16:27.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6171e-05 2023-11-01T04:16:27.929 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.929 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.929 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.930 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.6s6 47/31 ECSubWrite(tid=43, reqid=client.4559.0:351, at_version=47'6, trim_to=0'0, roll_forward_to=36'4))", 2023-11-01T04:16:27.930 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.248407+0000", 2023-11-01T04:16:27.930 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.50962747799999997, 2023-11-01T04:16:27.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202387496, 2023-11-01T04:16:27.930 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.930 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.931 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248407+0000", 2023-11-01T04:16:27.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248407+0000", 2023-11-01T04:16:27.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248408+0000", 2023-11-01T04:16:27.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.037e-06 2023-11-01T04:16:27.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248462+0000", 2023-11-01T04:16:27.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4497999999999999e-05 2023-11-01T04:16:27.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248501+0000", 2023-11-01T04:16:27.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8423000000000001e-05 2023-11-01T04:16:27.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.348639+0000", 2023-11-01T04:16:27.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100138353 2023-11-01T04:16:27.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.448915+0000", 2023-11-01T04:16:27.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100276063 2023-11-01T04:16:27.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.448949+0000", 2023-11-01T04:16:27.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3482000000000002e-05 2023-11-01T04:16:27.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.450761+0000", 2023-11-01T04:16:27.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001811785 2023-11-01T04:16:27.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.450794+0000", 2023-11-01T04:16:27.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3855e-05 2023-11-01T04:16:27.939 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.939 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.939 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.940 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.1s2 47/31 ECSubWrite(tid=42, reqid=client.4559.0:354, at_version=47'22, trim_to=0'0, roll_forward_to=46'20))", 2023-11-01T04:16:27.940 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.479712+0000", 2023-11-01T04:16:27.940 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.27832276299999997, 2023-11-01T04:16:27.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039588999999999996, 2023-11-01T04:16:27.941 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.941 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.941 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.479712+0000", 2023-11-01T04:16:27.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.479712+0000", 2023-11-01T04:16:27.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.479714+0000", 2023-11-01T04:16:27.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8700000000000001e-06 2023-11-01T04:16:27.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.480246+0000", 2023-11-01T04:16:27.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053202000000000002 2023-11-01T04:16:27.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.480274+0000", 2023-11-01T04:16:27.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8000999999999998e-05 2023-11-01T04:16:27.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.480292+0000", 2023-11-01T04:16:27.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8879e-05 2023-11-01T04:16:27.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.480390+0000", 2023-11-01T04:16:27.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7575e-05 2023-11-01T04:16:27.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.480405+0000", 2023-11-01T04:16:27.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.488e-05 2023-11-01T04:16:27.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.483634+0000", 2023-11-01T04:16:27.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032287259999999999 2023-11-01T04:16:27.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.483671+0000", 2023-11-01T04:16:27.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6949000000000001e-05 2023-11-01T04:16:27.949 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.949 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.950 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.950 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.1s2 47/31 ECSubWrite(tid=43, reqid=client.4559.0:355, at_version=47'23, trim_to=0'0, roll_forward_to=46'20))", 2023-11-01T04:16:27.950 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.484260+0000", 2023-11-01T04:16:27.950 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.27377398800000002, 2023-11-01T04:16:27.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102572046, 2023-11-01T04:16:27.951 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.951 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.951 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.484260+0000", 2023-11-01T04:16:27.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.484260+0000", 2023-11-01T04:16:27.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.484262+0000", 2023-11-01T04:16:27.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1170000000000001e-06 2023-11-01T04:16:27.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.484450+0000", 2023-11-01T04:16:27.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018824 2023-11-01T04:16:27.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.484468+0000", 2023-11-01T04:16:27.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8084000000000002e-05 2023-11-01T04:16:27.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.484476+0000", 2023-11-01T04:16:27.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2309999999999996e-06 2023-11-01T04:16:27.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.584699+0000", 2023-11-01T04:16:27.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10022244700000001 2023-11-01T04:16:27.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.584716+0000", 2023-11-01T04:16:27.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7042999999999999e-05 2023-11-01T04:16:27.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.586754+0000", 2023-11-01T04:16:27.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020380929999999999 2023-11-01T04:16:27.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.586832+0000", 2023-11-01T04:16:27.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8790999999999998e-05 2023-11-01T04:16:27.959 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.960 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.960 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.960 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.1s2 47/31 ECSubWrite(tid=44, reqid=client.4559.0:356, at_version=47'24, trim_to=0'0, roll_forward_to=46'20))", 2023-11-01T04:16:27.960 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.486733+0000", 2023-11-01T04:16:27.961 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.27130097399999997, 2023-11-01T04:16:27.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100697651, 2023-11-01T04:16:27.961 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:27.961 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:27.961 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:27.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:27.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.486733+0000", 2023-11-01T04:16:27.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:27.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.486733+0000", 2023-11-01T04:16:27.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:27.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:27.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.486734+0000", 2023-11-01T04:16:27.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0380000000000001e-06 2023-11-01T04:16:27.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:27.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.486756+0000", 2023-11-01T04:16:27.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1443000000000001e-05 2023-11-01T04:16:27.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:27.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.486774+0000", 2023-11-01T04:16:27.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8349999999999999e-05 2023-11-01T04:16:27.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:27.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.486788+0000", 2023-11-01T04:16:27.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3957e-05 2023-11-01T04:16:27.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:27.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.585678+0000", 2023-11-01T04:16:27.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.098890005000000003 2023-11-01T04:16:27.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:27.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.585694+0000", 2023-11-01T04:16:27.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5985999999999999e-05 2023-11-01T04:16:27.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:27.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.587389+0000", 2023-11-01T04:16:27.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016948130000000001 2023-11-01T04:16:27.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:27.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:27.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:27.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.587431+0000", 2023-11-01T04:16:27.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2058999999999998e-05 2023-11-01T04:16:27.970 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.970 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.970 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.970 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:27.970 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:27.970 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:27.972 INFO:tasks.rados.rados.0.smithi190.stdout:141: finishing write tid 1 to smithi19039307-50 2023-11-01T04:16:27.972 INFO:tasks.rados.rados.0.smithi190.stdout:141: finishing write tid 2 to smithi19039307-50 2023-11-01T04:16:27.972 INFO:tasks.rados.rados.0.smithi190.stdout:140: finishing write tid 4 to smithi19039307-31 2023-11-01T04:16:27.973 INFO:tasks.rados.rados.0.smithi190.stdout:140: oid 31 updating version 0 to 23 2023-11-01T04:16:27.973 INFO:tasks.rados.rados.0.smithi190.stdout:140: oid 31 updating version 23 to 24 2023-11-01T04:16:27.973 INFO:tasks.rados.rados.0.smithi190.stdout:140: oid 31 version 24 is already newer than 22 2023-11-01T04:16:27.973 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 31 v 24 (ObjNum 69 snap 7 seq_num 69) dirty exists 2023-11-01T04:16:27.973 INFO:tasks.rados.rados.0.smithi190.stdout:140: left oid 31 (ObjNum 69 snap 7 seq_num 69) 2023-11-01T04:16:27.973 INFO:tasks.rados.rados.0.smithi190.stdout:141: finishing write tid 3 to smithi19039307-50 2023-11-01T04:16:27.974 INFO:tasks.rados.rados.0.smithi190.stdout:139: finishing copy_from racing read to smithi19039307-27 2023-11-01T04:16:27.974 INFO:tasks.rados.rados.0.smithi190.stdout:142: finishing copy_from racing read to smithi19039307-19 2023-11-01T04:16:27.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-01T04:16:27.975 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:27.968+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:28.015 INFO:tasks.rados.rados.0.smithi190.stdout:141: finishing write tid 4 to smithi19039307-50 2023-11-01T04:16:28.016 INFO:tasks.rados.rados.0.smithi190.stdout:141: oid 50 updating version 0 to 13 2023-11-01T04:16:28.016 INFO:tasks.rados.rados.0.smithi190.stdout:141: oid 50 updating version 13 to 14 2023-11-01T04:16:28.016 INFO:tasks.rados.rados.0.smithi190.stdout:141: oid 50 updating version 14 to 15 2023-11-01T04:16:28.016 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 50 v 15 (ObjNum 70 snap 7 seq_num 70) dirty exists 2023-11-01T04:16:28.016 INFO:tasks.rados.rados.0.smithi190.stdout:141: left oid 50 (ObjNum 70 snap 7 seq_num 70) 2023-11-01T04:16:28.019 INFO:tasks.rados.rados.0.smithi190.stdout:139: finishing copy_from to smithi19039307-27 2023-11-01T04:16:28.020 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 27 v 22 (ObjNum 66 snap 6 seq_num 66) dirty exists 2023-11-01T04:16:28.034 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.8 to 1 (10000) 2023-11-01T04:16:28.041 INFO:tasks.rados.rados.0.smithi190.stdout:136: done (6 left) 2023-11-01T04:16:28.041 INFO:tasks.rados.rados.0.smithi190.stdout:139: done (5 left) 2023-11-01T04:16:28.041 INFO:tasks.rados.rados.0.smithi190.stdout:140: done (4 left) 2023-11-01T04:16:28.041 INFO:tasks.rados.rados.0.smithi190.stdout:141: done (3 left) 2023-11-01T04:16:28.041 INFO:tasks.rados.rados.0.smithi190.stdout:143: done (2 left) 2023-11-01T04:16:28.042 INFO:tasks.rados.rados.0.smithi190.stdout:144: done (1 left) 2023-11-01T04:16:28.042 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 42 current snap is 8 2023-11-01T04:16:28.042 INFO:tasks.rados.rados.0.smithi190.stdout:145: seq_num 71 ranges {704512=425984,1130496=573440,1703936=196608} 2023-11-01T04:16:28.047 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-11-01T04:16:28.054 INFO:tasks.rados.rados.0.smithi190.stdout:145: writing smithi19039307-42 from 704512 to 1130496 tid 1 2023-11-01T04:16:28.057 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2023-11-01T04:16:28.060 INFO:tasks.rados.rados.0.smithi190.stdout:145: writing smithi19039307-42 from 1130496 to 1703936 tid 2 2023-11-01T04:16:28.063 INFO:tasks.rados.rados.0.smithi190.stdout:145: writing smithi19039307-42 from 1703936 to 1900544 tid 3 2023-11-01T04:16:28.063 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 25 current snap is 8 2023-11-01T04:16:28.063 INFO:tasks.rados.rados.0.smithi190.stdout:146: seq_num 72 ranges {98304=442368,540672=720896,1261568=49152} 2023-11-01T04:16:28.069 INFO:tasks.rados.rados.0.smithi190.stdout:146: writing smithi19039307-25 from 98304 to 540672 tid 1 2023-11-01T04:16:28.073 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:28.071+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:28.078 INFO:tasks.rados.rados.0.smithi190.stdout:146: writing smithi19039307-25 from 540672 to 1261568 tid 2 2023-11-01T04:16:28.078 INFO:tasks.rados.rados.0.smithi190.stdout:146: writing smithi19039307-25 from 1261568 to 1310720 tid 3 2023-11-01T04:16:28.079 INFO:tasks.rados.rados.0.smithi190.stdout:147: rollback oid 18 current snap is 8 2023-11-01T04:16:28.079 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 18 to 2 2023-11-01T04:16:28.079 INFO:tasks.rados.rados.0.smithi190.stdout:148: snap_create 2023-11-01T04:16:28.122 INFO:tasks.rados.rados.0.smithi190.stdout:142: finishing copy_from to smithi19039307-19 2023-11-01T04:16:28.122 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 19 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-01T04:16:28.146 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:28.147 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:16:28.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.147 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:378 3.2s0 3.199c6df2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:16:28.147 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:28.080473+0000", 2023-11-01T04:16:28.147 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.066912685, 2023-11-01T04:16:28.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.066953056999999996, 2023-11-01T04:16:28.148 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.148 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting_for_map not empty", 2023-11-01T04:16:28.148 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.148 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:28.148 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:28.149 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 378 2023-11-01T04:16:28.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.149 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.080473+0000", 2023-11-01T04:16:28.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.080473+0000", 2023-11-01T04:16:28.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.080474+0000", 2023-11-01T04:16:28.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3340000000000001e-06 2023-11-01T04:16:28.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.080480+0000", 2023-11-01T04:16:28.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1700000000000002e-06 2023-11-01T04:16:28.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.080504+0000", 2023-11-01T04:16:28.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3509000000000001e-05 2023-11-01T04:16:28.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.080521+0000", 2023-11-01T04:16:28.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7473e-05 2023-11-01T04:16:28.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.080569+0000", 2023-11-01T04:16:28.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8013000000000003e-05 2023-11-01T04:16:28.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:28.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.080573+0000", 2023-11-01T04:16:28.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4920000000000002e-06 2023-11-01T04:16:28.156 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.157 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.157 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.157 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 48/31 ECSubWriteReply(tid=246, last_complete=47'15, committed=1, applied=1))", 2023-11-01T04:16:28.157 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:28.115353+0000", 2023-11-01T04:16:28.158 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.032032447999999998, 2023-11-01T04:16:28.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.032192762999999999, 2023-11-01T04:16:28.158 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.158 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:28.158 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.115353+0000", 2023-11-01T04:16:28.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.115353+0000", 2023-11-01T04:16:28.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.115355+0000", 2023-11-01T04:16:28.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4200000000000001e-06 2023-11-01T04:16:28.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.115370+0000", 2023-11-01T04:16:28.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.411e-05 2023-11-01T04:16:28.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.115402+0000", 2023-11-01T04:16:28.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2592000000000002e-05 2023-11-01T04:16:28.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.115424+0000", 2023-11-01T04:16:28.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1849e-05 2023-11-01T04:16:28.164 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.164 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.164 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.165 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:377 3.2s0 3.199c6df2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:16:28.165 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:28.080333+0000", 2023-11-01T04:16:28.165 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.067052125000000004, 2023-11-01T04:16:28.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.067290907999999997, 2023-11-01T04:16:28.165 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.165 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "op must wait for map", 2023-11-01T04:16:28.166 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.166 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:28.166 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:28.166 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 377 2023-11-01T04:16:28.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.167 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.080333+0000", 2023-11-01T04:16:28.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.080333+0000", 2023-11-01T04:16:28.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.080336+0000", 2023-11-01T04:16:28.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.678e-06 2023-11-01T04:16:28.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.080347+0000", 2023-11-01T04:16:28.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.145e-05 2023-11-01T04:16:28.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.080389+0000", 2023-11-01T04:16:28.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2061000000000003e-05 2023-11-01T04:16:28.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.080416+0000", 2023-11-01T04:16:28.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6307000000000001e-05 2023-11-01T04:16:28.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.080518+0000", 2023-11-01T04:16:28.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000102147 2023-11-01T04:16:28.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op must wait for map", 2023-11-01T04:16:28.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.080524+0000", 2023-11-01T04:16:28.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1580000000000002e-06 2023-11-01T04:16:28.173 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.173 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.174 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.174 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.174 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2023-11-01T04:16:28.174 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 3 2023-11-01T04:16:28.174 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:28.175 INFO:tasks.rados.rados.0.smithi190.stdout:145: finishing write tid 1 to smithi19039307-42 2023-11-01T04:16:28.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-01T04:16:28.176 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:28.175+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:28.276 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:28.272+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:28.302 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:28.303 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:28.303 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:16:28.303 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:16:28.303 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:28.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-01T04:16:28.367 INFO:tasks.rados.rados.0.smithi190.stdout:145: finishing write tid 2 to smithi19039307-42 2023-11-01T04:16:28.375 INFO:tasks.rados.rados.0.smithi190.stdout:147: finishing rollback tid 1 to smithi19039307-18 2023-11-01T04:16:28.378 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:28.375+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:28.438 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:28.438 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:16:28.438 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:16:28.439 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:16:28.439 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.439 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:28.439 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138478+0000", 2023-11-01T04:16:28.439 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 23.297111267999998, 2023-11-01T04:16:28.440 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51616329400000005, 2023-11-01T04:16:28.440 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.440 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:28.440 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.440 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:28.440 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:28.441 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 111 2023-11-01T04:16:28.441 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.441 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.441 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138478+0000", 2023-11-01T04:16:28.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.442 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.443 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138478+0000", 2023-11-01T04:16:28.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.443 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.443 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.443 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.443 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138479+0000", 2023-11-01T04:16:28.444 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6000000000000003e-07 2023-11-01T04:16:28.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.444 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.444 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.444 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138484+0000", 2023-11-01T04:16:28.445 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5099999999999998e-06 2023-11-01T04:16:28.445 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.445 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.445 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.445 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138494+0000", 2023-11-01T04:16:28.445 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3659999999999998e-06 2023-11-01T04:16:28.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.446 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.446 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.446 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138501+0000", 2023-11-01T04:16:28.446 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7230000000000004e-06 2023-11-01T04:16:28.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.447 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.243922+0000", 2023-11-01T04:16:28.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105420454 2023-11-01T04:16:28.447 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.448 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.244148+0000", 2023-11-01T04:16:28.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022663800000000001 2023-11-01T04:16:28.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:28.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.244505+0000", 2023-11-01T04:16:28.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00035644499999999999 2023-11-01T04:16:28.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:28.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.250339+0000", 2023-11-01T04:16:28.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058344959999999998 2023-11-01T04:16:28.450 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:28.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.654434+0000", 2023-11-01T04:16:28.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40409476900000002 2023-11-01T04:16:28.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:28.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.654518+0000", 2023-11-01T04:16:28.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4421999999999999e-05 2023-11-01T04:16:28.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.654641+0000", 2023-11-01T04:16:28.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012281099999999999 2023-11-01T04:16:28.453 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.453 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.453 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.454 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:28.454 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138513+0000", 2023-11-01T04:16:28.454 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 23.29707668, 2023-11-01T04:16:28.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51640948099999995, 2023-11-01T04:16:28.454 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.455 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:28.455 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.455 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:28.455 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:28.455 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 112 2023-11-01T04:16:28.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.456 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138513+0000", 2023-11-01T04:16:28.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138513+0000", 2023-11-01T04:16:28.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.458 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.458 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138513+0000", 2023-11-01T04:16:28.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5000000000000002e-07 2023-11-01T04:16:28.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138680+0000", 2023-11-01T04:16:28.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016725400000000001 2023-11-01T04:16:28.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138694+0000", 2023-11-01T04:16:28.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3495999999999999e-05 2023-11-01T04:16:28.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138706+0000", 2023-11-01T04:16:28.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1557e-05 2023-11-01T04:16:28.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.245369+0000", 2023-11-01T04:16:28.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10666384600000001 2023-11-01T04:16:28.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.245509+0000", 2023-11-01T04:16:28.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014007900000000001 2023-11-01T04:16:28.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.463 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:28.463 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.245728+0000", 2023-11-01T04:16:28.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000218345 2023-11-01T04:16:28.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:28.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352473+0000", 2023-11-01T04:16:28.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.106745693 2023-11-01T04:16:28.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:28.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.654776+0000", 2023-11-01T04:16:28.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30230230200000002 2023-11-01T04:16:28.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:28.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.654863+0000", 2023-11-01T04:16:28.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7638000000000004e-05 2023-11-01T04:16:28.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.654922+0000", 2023-11-01T04:16:28.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8621e-05 2023-11-01T04:16:28.467 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.467 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.468 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.468 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:28.468 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138723+0000", 2023-11-01T04:16:28.468 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 23.296866584, 2023-11-01T04:16:28.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51647989999999999, 2023-11-01T04:16:28.469 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.469 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:28.469 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.469 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:28.470 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:28.470 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 113 2023-11-01T04:16:28.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.470 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138723+0000", 2023-11-01T04:16:28.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138723+0000", 2023-11-01T04:16:28.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138724+0000", 2023-11-01T04:16:28.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5600000000000004e-07 2023-11-01T04:16:28.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138729+0000", 2023-11-01T04:16:28.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7799999999999997e-06 2023-11-01T04:16:28.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138741+0000", 2023-11-01T04:16:28.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1846e-05 2023-11-01T04:16:28.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138749+0000", 2023-11-01T04:16:28.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0499999999999992e-06 2023-11-01T04:16:28.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.246597+0000", 2023-11-01T04:16:28.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10784767300000001 2023-11-01T04:16:28.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.246820+0000", 2023-11-01T04:16:28.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022339199999999999 2023-11-01T04:16:28.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:28.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.247168+0000", 2023-11-01T04:16:28.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000347521 2023-11-01T04:16:28.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:28.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352480+0000", 2023-11-01T04:16:28.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105312482 2023-11-01T04:16:28.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:28.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.655031+0000", 2023-11-01T04:16:28.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30255108200000003 2023-11-01T04:16:28.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:28.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.655110+0000", 2023-11-01T04:16:28.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8678999999999997e-05 2023-11-01T04:16:28.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.655203+0000", 2023-11-01T04:16:28.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2439e-05 2023-11-01T04:16:28.482 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.482 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.482 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.483 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:28.483 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138761+0000", 2023-11-01T04:16:28.483 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 23.296828012999999, 2023-11-01T04:16:28.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61681865999999996, 2023-11-01T04:16:28.483 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.483 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:28.484 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.484 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:28.484 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:28.484 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 114 2023-11-01T04:16:28.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.484 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138761+0000", 2023-11-01T04:16:28.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138761+0000", 2023-11-01T04:16:28.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138762+0000", 2023-11-01T04:16:28.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4300000000000003e-07 2023-11-01T04:16:28.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138768+0000", 2023-11-01T04:16:28.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7729999999999996e-06 2023-11-01T04:16:28.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138777+0000", 2023-11-01T04:16:28.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1929999999999999e-06 2023-11-01T04:16:28.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138784+0000", 2023-11-01T04:16:28.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3830000000000003e-06 2023-11-01T04:16:28.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.248094+0000", 2023-11-01T04:16:28.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10930951699999999 2023-11-01T04:16:28.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.248210+0000", 2023-11-01T04:16:28.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011579299999999999 2023-11-01T04:16:28.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:28.492 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.248407+0000", 2023-11-01T04:16:28.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019723000000000001 2023-11-01T04:16:28.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:28.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352490+0000", 2023-11-01T04:16:28.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104083393 2023-11-01T04:16:28.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:28.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755390+0000", 2023-11-01T04:16:28.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40290009500000001 2023-11-01T04:16:28.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:28.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755474+0000", 2023-11-01T04:16:28.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3323000000000002e-05 2023-11-01T04:16:28.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755580+0000", 2023-11-01T04:16:28.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010631700000000001 2023-11-01T04:16:28.496 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.496 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.496 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.497 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:28.497 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138795+0000", 2023-11-01T04:16:28.497 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 23.296793855000001, 2023-11-01T04:16:28.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61707305000000001, 2023-11-01T04:16:28.497 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.498 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:28.498 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.498 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:28.498 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:28.498 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 115 2023-11-01T04:16:28.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.499 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138795+0000", 2023-11-01T04:16:28.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138795+0000", 2023-11-01T04:16:28.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138796+0000", 2023-11-01T04:16:28.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3300000000000002e-07 2023-11-01T04:16:28.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138931+0000", 2023-11-01T04:16:28.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013525000000000001 2023-11-01T04:16:28.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138947+0000", 2023-11-01T04:16:28.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5478999999999999e-05 2023-11-01T04:16:28.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138956+0000", 2023-11-01T04:16:28.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1530000000000003e-06 2023-11-01T04:16:28.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.249005+0000", 2023-11-01T04:16:28.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.110048914 2023-11-01T04:16:28.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.249150+0000", 2023-11-01T04:16:28.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014491600000000001 2023-11-01T04:16:28.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:28.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.249351+0000", 2023-11-01T04:16:28.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000201516 2023-11-01T04:16:28.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:28.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352738+0000", 2023-11-01T04:16:28.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10338632 2023-11-01T04:16:28.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:28.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755668+0000", 2023-11-01T04:16:28.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40293035700000002 2023-11-01T04:16:28.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:28.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755765+0000", 2023-11-01T04:16:28.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7081000000000001e-05 2023-11-01T04:16:28.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755868+0000", 2023-11-01T04:16:28.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010343100000000001 2023-11-01T04:16:28.510 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.510 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.510 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.511 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:28.511 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138971+0000", 2023-11-01T04:16:28.511 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 23.296618317, 2023-11-01T04:16:28.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71733164100000002, 2023-11-01T04:16:28.513 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.513 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:28.513 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.513 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:28.514 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:28.514 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 116 2023-11-01T04:16:28.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.514 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138971+0000", 2023-11-01T04:16:28.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138971+0000", 2023-11-01T04:16:28.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138972+0000", 2023-11-01T04:16:28.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4000000000000001e-07 2023-11-01T04:16:28.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138988+0000", 2023-11-01T04:16:28.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6354999999999998e-05 2023-11-01T04:16:28.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138998+0000", 2023-11-01T04:16:28.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0285999999999999e-05 2023-11-01T04:16:28.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139010+0000", 2023-11-01T04:16:28.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1807000000000001e-05 2023-11-01T04:16:28.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.350520+0000", 2023-11-01T04:16:28.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21151004200000001 2023-11-01T04:16:28.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.350668+0000", 2023-11-01T04:16:28.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014824900000000001 2023-11-01T04:16:28.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:28.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.350873+0000", 2023-11-01T04:16:28.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020507899999999999 2023-11-01T04:16:28.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:28.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352744+0000", 2023-11-01T04:16:28.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001870741 2023-11-01T04:16:28.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:28.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856134+0000", 2023-11-01T04:16:28.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50339015399999998 2023-11-01T04:16:28.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:28.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856235+0000", 2023-11-01T04:16:28.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010087200000000001 2023-11-01T04:16:28.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.525 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856303+0000", 2023-11-01T04:16:28.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7316000000000004e-05 2023-11-01T04:16:28.526 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.526 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.526 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.527 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:28.527 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.139035+0000", 2023-11-01T04:16:28.527 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 23.296554330999999, 2023-11-01T04:16:28.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71770837499999995, 2023-11-01T04:16:28.527 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.527 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:28.528 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.528 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:28.528 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:28.528 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 117 2023-11-01T04:16:28.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.529 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139035+0000", 2023-11-01T04:16:28.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139035+0000", 2023-11-01T04:16:28.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139036+0000", 2023-11-01T04:16:28.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7999999999999995e-07 2023-11-01T04:16:28.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139042+0000", 2023-11-01T04:16:28.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9499999999999998e-06 2023-11-01T04:16:28.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139051+0000", 2023-11-01T04:16:28.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7489999999999993e-06 2023-11-01T04:16:28.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139059+0000", 2023-11-01T04:16:28.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8159999999999994e-06 2023-11-01T04:16:28.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.351436+0000", 2023-11-01T04:16:28.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21237641400000001 2023-11-01T04:16:28.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.351583+0000", 2023-11-01T04:16:28.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000147186 2023-11-01T04:16:28.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:28.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.351772+0000", 2023-11-01T04:16:28.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018879300000000001 2023-11-01T04:16:28.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:28.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.553963+0000", 2023-11-01T04:16:28.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20219130599999999 2023-11-01T04:16:28.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:28.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856458+0000", 2023-11-01T04:16:28.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30249546300000002 2023-11-01T04:16:28.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:28.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856632+0000", 2023-11-01T04:16:28.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000173611 2023-11-01T04:16:28.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856743+0000", 2023-11-01T04:16:28.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111407 2023-11-01T04:16:28.540 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.540 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.540 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.541 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:131 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:28.541 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.621413+0000", 2023-11-01T04:16:28.541 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 21.814175885000001, 2023-11-01T04:16:28.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50410262299999997, 2023-11-01T04:16:28.541 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.542 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:28.542 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.542 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:28.542 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:28.542 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 131 2023-11-01T04:16:28.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.543 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621413+0000", 2023-11-01T04:16:28.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621413+0000", 2023-11-01T04:16:28.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621415+0000", 2023-11-01T04:16:28.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2699999999999999e-06 2023-11-01T04:16:28.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621583+0000", 2023-11-01T04:16:28.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016864500000000001 2023-11-01T04:16:28.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621598+0000", 2023-11-01T04:16:28.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4262e-05 2023-11-01T04:16:28.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.621607+0000", 2023-11-01T04:16:28.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9029999999999999e-06 2023-11-01T04:16:28.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622526+0000", 2023-11-01T04:16:28.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091876800000000004 2023-11-01T04:16:28.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622628+0000", 2023-11-01T04:16:28.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101413 2023-11-01T04:16:28.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:28.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622761+0000", 2023-11-01T04:16:28.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000133781 2023-11-01T04:16:28.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:28.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835742+0000", 2023-11-01T04:16:28.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21298055900000001 2023-11-01T04:16:28.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:28.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.125351+0000", 2023-11-01T04:16:28.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28960872700000001 2023-11-01T04:16:28.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:28.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.125448+0000", 2023-11-01T04:16:28.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6947999999999994e-05 2023-11-01T04:16:28.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.125516+0000", 2023-11-01T04:16:28.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8347000000000001e-05 2023-11-01T04:16:28.554 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.554 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.554 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.555 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:28.555 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622325+0000", 2023-11-01T04:16:28.555 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 21.813264643, 2023-11-01T04:16:28.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50422977499999999, 2023-11-01T04:16:28.555 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.556 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:28.556 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.556 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:28.556 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:28.556 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 136 2023-11-01T04:16:28.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.557 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622325+0000", 2023-11-01T04:16:28.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622325+0000", 2023-11-01T04:16:28.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622325+0000", 2023-11-01T04:16:28.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7400000000000003e-07 2023-11-01T04:16:28.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622330+0000", 2023-11-01T04:16:28.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8720000000000001e-06 2023-11-01T04:16:28.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622340+0000", 2023-11-01T04:16:28.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6739999999999992e-06 2023-11-01T04:16:28.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622347+0000", 2023-11-01T04:16:28.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3660000000000003e-06 2023-11-01T04:16:28.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.725754+0000", 2023-11-01T04:16:28.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.103406496 2023-11-01T04:16:28.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.726001+0000", 2023-11-01T04:16:28.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024736099999999998 2023-11-01T04:16:28.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:28.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.726371+0000", 2023-11-01T04:16:28.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00037005099999999998 2023-11-01T04:16:28.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:28.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835786+0000", 2023-11-01T04:16:28.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.109414463 2023-11-01T04:16:28.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:28.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126424+0000", 2023-11-01T04:16:28.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29063866999999999 2023-11-01T04:16:28.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:28.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126500+0000", 2023-11-01T04:16:28.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6212999999999997e-05 2023-11-01T04:16:28.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126554+0000", 2023-11-01T04:16:28.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3934999999999998e-05 2023-11-01T04:16:28.568 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.568 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.568 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.568 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:28.569 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622359+0000", 2023-11-01T04:16:28.569 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 21.813230707999999, 2023-11-01T04:16:28.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50439429300000005, 2023-11-01T04:16:28.569 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.569 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:28.569 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.570 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:28.570 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:28.570 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 137 2023-11-01T04:16:28.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.570 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622359+0000", 2023-11-01T04:16:28.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622359+0000", 2023-11-01T04:16:28.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622359+0000", 2023-11-01T04:16:28.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.44e-07 2023-11-01T04:16:28.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.573 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-01T04:16:28.575 INFO:tasks.rados.rados.0.smithi190.stdout:145: finishing write tid 3 to smithi19039307-42 2023-11-01T04:16:28.576 INFO:tasks.rados.rados.0.smithi190.stdout:145: finishing write tid 4 to smithi19039307-42 2023-11-01T04:16:28.576 INFO:tasks.rados.rados.0.smithi190.stdout:147: finishing rollback tid 2 to smithi19039307-18 2023-11-01T04:16:28.576 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 18 v 39 (ObjNum 43 snap 0 seq_num 43) dirty exists 2023-11-01T04:16:28.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622516+0000", 2023-11-01T04:16:28.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000156945 2023-11-01T04:16:28.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622547+0000", 2023-11-01T04:16:28.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0747999999999998e-05 2023-11-01T04:16:28.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622564+0000", 2023-11-01T04:16:28.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7516e-05 2023-11-01T04:16:28.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.727341+0000", 2023-11-01T04:16:28.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104776878 2023-11-01T04:16:28.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.727568+0000", 2023-11-01T04:16:28.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000226908 2023-11-01T04:16:28.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:28.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.727955+0000", 2023-11-01T04:16:28.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038721900000000001 2023-11-01T04:16:28.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:28.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835801+0000", 2023-11-01T04:16:28.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.107845343 2023-11-01T04:16:28.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:28.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126619+0000", 2023-11-01T04:16:28.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.290817879 2023-11-01T04:16:28.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:28.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126696+0000", 2023-11-01T04:16:28.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7571999999999995e-05 2023-11-01T04:16:28.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.126753+0000", 2023-11-01T04:16:28.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6641000000000002e-05 2023-11-01T04:16:28.585 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.585 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.585 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.586 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:28.586 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622584+0000", 2023-11-01T04:16:28.586 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 21.813004880000001, 2023-11-01T04:16:28.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50666713100000005, 2023-11-01T04:16:28.587 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.587 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:28.587 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.587 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:28.587 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:28.588 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 138 2023-11-01T04:16:28.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.588 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622584+0000", 2023-11-01T04:16:28.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622584+0000", 2023-11-01T04:16:28.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622585+0000", 2023-11-01T04:16:28.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0640000000000001e-06 2023-11-01T04:16:28.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622591+0000", 2023-11-01T04:16:28.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9059999999999999e-06 2023-11-01T04:16:28.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622602+0000", 2023-11-01T04:16:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0309000000000001e-05 2023-11-01T04:16:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622609+0000", 2023-11-01T04:16:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4869999999999996e-06 2023-11-01T04:16:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.729780+0000", 2023-11-01T04:16:28.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10717114799999999 2023-11-01T04:16:28.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.730026+0000", 2023-11-01T04:16:28.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024591700000000001 2023-11-01T04:16:28.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:28.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.730386+0000", 2023-11-01T04:16:28.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036019099999999999 2023-11-01T04:16:28.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:28.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835810+0000", 2023-11-01T04:16:28.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105423431 2023-11-01T04:16:28.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:28.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.129130+0000", 2023-11-01T04:16:28.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29331975399999999 2023-11-01T04:16:28.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:28.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.129203+0000", 2023-11-01T04:16:28.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3418999999999996e-05 2023-11-01T04:16:28.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.129252+0000", 2023-11-01T04:16:28.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8504999999999997e-05 2023-11-01T04:16:28.599 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.599 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.599 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.600 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:28.600 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622627+0000", 2023-11-01T04:16:28.600 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 21.812962632000001, 2023-11-01T04:16:28.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60704219400000003, 2023-11-01T04:16:28.600 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.600 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:28.601 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.601 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:28.601 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:28.601 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 139 2023-11-01T04:16:28.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.601 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622627+0000", 2023-11-01T04:16:28.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622627+0000", 2023-11-01T04:16:28.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622627+0000", 2023-11-01T04:16:28.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5400000000000001e-07 2023-11-01T04:16:28.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622632+0000", 2023-11-01T04:16:28.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0420000000000002e-06 2023-11-01T04:16:28.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622642+0000", 2023-11-01T04:16:28.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.713e-06 2023-11-01T04:16:28.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622649+0000", 2023-11-01T04:16:28.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4460000000000004e-06 2023-11-01T04:16:28.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.831442+0000", 2023-11-01T04:16:28.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20879259 2023-11-01T04:16:28.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.831689+0000", 2023-11-01T04:16:28.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000247087 2023-11-01T04:16:28.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:28.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.832051+0000", 2023-11-01T04:16:28.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036175699999999999 2023-11-01T04:16:28.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:28.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.837325+0000", 2023-11-01T04:16:28.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0052741289999999998 2023-11-01T04:16:28.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:28.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229409+0000", 2023-11-01T04:16:28.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39208416000000001 2023-11-01T04:16:28.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:28.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229566+0000", 2023-11-01T04:16:28.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015727499999999999 2023-11-01T04:16:28.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229669+0000", 2023-11-01T04:16:28.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000102341 2023-11-01T04:16:28.613 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.613 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.613 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.613 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:28.614 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622661+0000", 2023-11-01T04:16:28.614 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 21.812928240000002, 2023-11-01T04:16:28.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60735416200000003, 2023-11-01T04:16:28.614 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.614 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:28.614 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.615 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:28.615 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:28.615 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 140 2023-11-01T04:16:28.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.615 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622661+0000", 2023-11-01T04:16:28.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622661+0000", 2023-11-01T04:16:28.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622662+0000", 2023-11-01T04:16:28.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3e-07 2023-11-01T04:16:28.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622803+0000", 2023-11-01T04:16:28.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014142199999999999 2023-11-01T04:16:28.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622816+0000", 2023-11-01T04:16:28.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2683e-05 2023-11-01T04:16:28.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622825+0000", 2023-11-01T04:16:28.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9959999999999999e-06 2023-11-01T04:16:28.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.833052+0000", 2023-11-01T04:16:28.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21022711699999999 2023-11-01T04:16:28.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.833201+0000", 2023-11-01T04:16:28.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000149059 2023-11-01T04:16:28.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:28.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.833405+0000", 2023-11-01T04:16:28.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020363600000000001 2023-11-01T04:16:28.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:28.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.837343+0000", 2023-11-01T04:16:28.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039379590000000004 2023-11-01T04:16:28.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:28.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229871+0000", 2023-11-01T04:16:28.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39252865399999998 2023-11-01T04:16:28.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:28.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229957+0000", 2023-11-01T04:16:28.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5962000000000001e-05 2023-11-01T04:16:28.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.230015+0000", 2023-11-01T04:16:28.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8044000000000002e-05 2023-11-01T04:16:28.626 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.626 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.627 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.627 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:28.627 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622843+0000", 2023-11-01T04:16:28.627 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 21.812745914000001, 2023-11-01T04:16:28.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70733220600000002, 2023-11-01T04:16:28.628 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.628 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:28.628 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.628 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:28.628 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:28.629 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 141 2023-11-01T04:16:28.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.629 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622843+0000", 2023-11-01T04:16:28.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622843+0000", 2023-11-01T04:16:28.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622844+0000", 2023-11-01T04:16:28.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7400000000000002e-07 2023-11-01T04:16:28.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622850+0000", 2023-11-01T04:16:28.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4330000000000003e-06 2023-11-01T04:16:28.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622859+0000", 2023-11-01T04:16:28.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5829999999999996e-06 2023-11-01T04:16:28.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.722958+0000", 2023-11-01T04:16:28.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100098824 2023-11-01T04:16:28.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835872+0000", 2023-11-01T04:16:28.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.112913586 2023-11-01T04:16:28.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.836107+0000", 2023-11-01T04:16:28.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023507399999999999 2023-11-01T04:16:28.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:28.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.836474+0000", 2023-11-01T04:16:28.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036769800000000002 2023-11-01T04:16:28.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:28.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.837993+0000", 2023-11-01T04:16:28.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001518192 2023-11-01T04:16:28.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:28.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.329913+0000", 2023-11-01T04:16:28.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.49192055499999998 2023-11-01T04:16:28.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:28.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.330067+0000", 2023-11-01T04:16:28.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000154012 2023-11-01T04:16:28.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.330176+0000", 2023-11-01T04:16:28.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010847499999999999 2023-11-01T04:16:28.640 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.640 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.640 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.641 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:207 3.2s0 3:42ba522a:::smithi19039307-20:head [copy-from ver 4] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:28.641 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.965579+0000", 2023-11-01T04:16:28.641 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 11.470009919000001, 2023-11-01T04:16:28.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62034388299999998, 2023-11-01T04:16:28.641 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.642 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:28.642 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.642 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:28.642 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:28.642 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 207 2023-11-01T04:16:28.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.643 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965579+0000", 2023-11-01T04:16:28.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965579+0000", 2023-11-01T04:16:28.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965581+0000", 2023-11-01T04:16:28.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1969999999999999e-06 2023-11-01T04:16:28.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965617+0000", 2023-11-01T04:16:28.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6100999999999999e-05 2023-11-01T04:16:28.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965640+0000", 2023-11-01T04:16:28.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3566000000000001e-05 2023-11-01T04:16:28.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965657+0000", 2023-11-01T04:16:28.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6413e-05 2023-11-01T04:16:28.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.066185+0000", 2023-11-01T04:16:28.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100528453 2023-11-01T04:16:28.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.066331+0000", 2023-11-01T04:16:28.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014583099999999999 2023-11-01T04:16:28.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:28.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.381227+0000", 2023-11-01T04:16:28.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31489625199999999 2023-11-01T04:16:28.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:28.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.384145+0000", 2023-11-01T04:16:28.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029175310000000001 2023-11-01T04:16:28.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:28.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.585865+0000", 2023-11-01T04:16:28.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.201720436 2023-11-01T04:16:28.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.585923+0000", 2023-11-01T04:16:28.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8103000000000003e-05 2023-11-01T04:16:28.653 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.653 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.653 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.653 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:209 3.3s0 3:ca2966a4:::smithi19039307-37:head [read 0~606208,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:28.654 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.966335+0000", 2023-11-01T04:16:28.654 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 11.469254059000001, 2023-11-01T04:16:28.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61218852199999996, 2023-11-01T04:16:28.654 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.654 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:28.654 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.655 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:28.655 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:28.655 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 209 2023-11-01T04:16:28.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.655 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966335+0000", 2023-11-01T04:16:28.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966335+0000", 2023-11-01T04:16:28.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966337+0000", 2023-11-01T04:16:28.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1430000000000001e-06 2023-11-01T04:16:28.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966349+0000", 2023-11-01T04:16:28.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2053e-05 2023-11-01T04:16:28.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966385+0000", 2023-11-01T04:16:28.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5438999999999997e-05 2023-11-01T04:16:28.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966408+0000", 2023-11-01T04:16:28.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3349e-05 2023-11-01T04:16:28.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970845+0000", 2023-11-01T04:16:28.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044366639999999999 2023-11-01T04:16:28.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970977+0000", 2023-11-01T04:16:28.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013261100000000001 2023-11-01T04:16:28.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.578524+0000", 2023-11-01T04:16:28.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60754626300000003 2023-11-01T04:16:28.663 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.663 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.663 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.664 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:210 3.3s0 3:ca2966a4:::smithi19039307-37:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:28.664 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.966473+0000", 2023-11-01T04:16:28.664 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 11.469116372, 2023-11-01T04:16:28.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61269195099999996, 2023-11-01T04:16:28.664 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.665 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:28.665 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.665 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:28.665 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:28.665 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 210 2023-11-01T04:16:28.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.666 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966473+0000", 2023-11-01T04:16:28.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966473+0000", 2023-11-01T04:16:28.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966474+0000", 2023-11-01T04:16:28.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4500000000000001e-06 2023-11-01T04:16:28.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966484+0000", 2023-11-01T04:16:28.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2269999999999999e-06 2023-11-01T04:16:28.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966508+0000", 2023-11-01T04:16:28.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4596e-05 2023-11-01T04:16:28.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966526+0000", 2023-11-01T04:16:28.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7938999999999999e-05 2023-11-01T04:16:28.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.071473+0000", 2023-11-01T04:16:28.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10494663799999999 2023-11-01T04:16:28.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.071640+0000", 2023-11-01T04:16:28.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016699 2023-11-01T04:16:28.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.579165+0000", 2023-11-01T04:16:28.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50752511099999997 2023-11-01T04:16:28.673 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.673 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.673 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.674 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:211 3.3s0 3:ca2966a4:::smithi19039307-37:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:28.674 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.966567+0000", 2023-11-01T04:16:28.674 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 11.469022734999999, 2023-11-01T04:16:28.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61304655900000005, 2023-11-01T04:16:28.675 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.675 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:28.675 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.675 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:28.675 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:28.675 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 211 2023-11-01T04:16:28.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.676 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966567+0000", 2023-11-01T04:16:28.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966567+0000", 2023-11-01T04:16:28.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966568+0000", 2023-11-01T04:16:28.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2529999999999999e-06 2023-11-01T04:16:28.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966575+0000", 2023-11-01T04:16:28.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4030000000000001e-06 2023-11-01T04:16:28.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966595+0000", 2023-11-01T04:16:28.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0208999999999999e-05 2023-11-01T04:16:28.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966611+0000", 2023-11-01T04:16:28.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6082999999999998e-05 2023-11-01T04:16:28.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.072213+0000", 2023-11-01T04:16:28.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105601509 2023-11-01T04:16:28.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.072432+0000", 2023-11-01T04:16:28.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000218724 2023-11-01T04:16:28.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.579613+0000", 2023-11-01T04:16:28.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50718137799999996 2023-11-01T04:16:28.684 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.684 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.684 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.684 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:212 3.3s0 3:c87735c0:::smithi19039307-50:head [create,setxattr XNeZS[lOPtVca\\Pa[\\rdfbinb[bcjhtYSpmcafZm (1928) in=40b,setxattr TfflYYqPnps]lo^qSlp___TitNdd`]kaPTVOTcsh (1648) in=40b,setxattr TNb_onpY[j_dkhfmnWtm]Yktg]]RNtbQe[OUTNlX (770) in=40b,setxattr ZcgnkhY\\orTYaZh]PPldj`SNq`p\\cX`k]h\\bZY_Y (700) in=40b,setxattr gPQVnqhpsVhW_kq`QnafXrlRZtZ\\YnRftgQTishR (595) in=40b,setxattr U`XqYi[WhoqnpSPRc\\RhnppZp^qY\\qUfjpqs[r^a (1892) in=40b,setxattr cTmhd`pWgrRf_VfVgOtgeOP^ShP]lsOtpoTj[\\gl (418) in=40b,setxattr aa_taYTfslg`lRRrOkkraN\\XiW^V\\ljl]NqXin`a (314) in=40b,setxattr ibl]`igtgVThY^Qse\\okYYUtTmolX\\W]p]Q[P\\il (1403) in=40b,setxattr \\ZabSO[UUqeZa`hddNh[ZjaRRgUZoQtkn\\QdsX]W (1541) in=40b,setxattr jUkQUPTNbo`rZNmYlUmhb]ojc]aeheoVRfp]bPWf (5052) in=40b,setxattr Ofj`oQ^cR]da^SPsTaXmmPse\\OZ`VSqQ]U`efsZl (419) in=40b,setxattr a[cdkWos\\OrUVPciUaa^_^`lWVTScffiWkd\\[kgf (1261) in=40b,setxattr jXb]\\WhbatdZ[qTTiZO]kPd^`l[kajOYiQ[iqSYc (244) in=40b,setxattr \\im]jQhbanUn\\_^h`jaU]W`ReaZN_je[]kgg (1578) in=36b,setxattr ZbosoRORS^_de]rN^_daOiha\\Q\\eRae^[sQUjXsV (9120) in=40b,setxattr gcn]XpZUkOZr\\pkrVVZpYiR\\alh_PaZa_S[PYcn] (384) in=40b,setxattr \\^qUOaNXSTjtmafhfbo\\gseTTkV`mgnacYq^bkcc (34) in=40b,setxattr iUa^mOQ]TqjVSUU_nXfY]f`NjljiiaYbrm[o]aes (1611) in=40b,setxattr tXaQPQbc^aiV[^reVOPpib^caoWf^XgqilO^_g[k (1016) in=40b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:28.685 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.970354+0000", 2023-11-01T04:16:28.685 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 11.465235126, 2023-11-01T04:16:28.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91502122100000005, 2023-11-01T04:16:28.685 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.685 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:28.685 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.686 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:28.686 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:28.686 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 212 2023-11-01T04:16:28.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.686 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970354+0000", 2023-11-01T04:16:28.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970354+0000", 2023-11-01T04:16:28.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970356+0000", 2023-11-01T04:16:28.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3070000000000001e-06 2023-11-01T04:16:28.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970468+0000", 2023-11-01T04:16:28.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111496 2023-11-01T04:16:28.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970492+0000", 2023-11-01T04:16:28.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4096000000000001e-05 2023-11-01T04:16:28.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970511+0000", 2023-11-01T04:16:28.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9069e-05 2023-11-01T04:16:28.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.579670+0000", 2023-11-01T04:16:28.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60915843300000005 2023-11-01T04:16:28.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.579779+0000", 2023-11-01T04:16:28.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000109417 2023-11-01T04:16:28.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:28.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.580741+0000", 2023-11-01T04:16:28.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00096157299999999996 2023-11-01T04:16:28.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:28.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.582542+0000", 2023-11-01T04:16:28.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018016130000000001 2023-11-01T04:16:28.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:28.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.885257+0000", 2023-11-01T04:16:28.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30271534300000003 2023-11-01T04:16:28.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.885375+0000", 2023-11-01T04:16:28.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117874 2023-11-01T04:16:28.696 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.697 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.697 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.697 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:361 3.3s0 3:c87735c0:::smithi19039307-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:28.697 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.511259+0000", 2023-11-01T04:16:28.698 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.924330294, 2023-11-01T04:16:28.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50546745000000004, 2023-11-01T04:16:28.698 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:28.698 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:28.698 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:28.698 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:28.699 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:28.699 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 361 2023-11-01T04:16:28.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.699 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:28.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:28.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.511259+0000", 2023-11-01T04:16:28.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:28.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.511259+0000", 2023-11-01T04:16:28.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:28.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:28.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.511259+0000", 2023-11-01T04:16:28.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.06e-07 2023-11-01T04:16:28.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:28.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.511262+0000", 2023-11-01T04:16:28.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8650000000000001e-06 2023-11-01T04:16:28.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:28.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.511277+0000", 2023-11-01T04:16:28.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4893e-05 2023-11-01T04:16:28.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:28.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.511284+0000", 2023-11-01T04:16:28.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3180000000000003e-06 2023-11-01T04:16:28.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.513454+0000", 2023-11-01T04:16:28.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021702620000000001 2023-11-01T04:16:28.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2023-11-01T04:16:28.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.513542+0000", 2023-11-01T04:16:28.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7729e-05 2023-11-01T04:16:28.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:28.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.914456+0000", 2023-11-01T04:16:28.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.400914206 2023-11-01T04:16:28.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:28.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.914736+0000", 2023-11-01T04:16:28.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00028035799999999997 2023-11-01T04:16:28.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:28.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:28.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:28.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.016726+0000", 2023-11-01T04:16:28.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101990313 2023-11-01T04:16:28.708 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.709 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.709 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.709 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:28.709 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:28.709 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:28.710 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-01T04:16:28.710 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:28.574+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:28.712 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:28.708+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:28.712 INFO:tasks.rados.rados.0.smithi190.stdout:146: finishing write tid 1 to smithi19039307-25 2023-11-01T04:16:28.712 INFO:tasks.rados.rados.0.smithi190.stdout:146: finishing write tid 2 to smithi19039307-25 2023-11-01T04:16:28.712 INFO:tasks.rados.rados.0.smithi190.stdout:146: finishing write tid 3 to smithi19039307-25 2023-11-01T04:16:28.713 INFO:tasks.rados.rados.0.smithi190.stdout:146: finishing write tid 4 to smithi19039307-25 2023-11-01T04:16:28.725 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:28.725 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-11-01T04:16:28.745 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:28.746 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-01T04:16:28.764 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:28.764 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-01T04:16:28.770 INFO:tasks.rados.rados.0.smithi190.stdout:145: finishing write tid 5 to smithi19039307-42 2023-11-01T04:16:28.770 INFO:tasks.rados.rados.0.smithi190.stdout:145: oid 42 updating version 0 to 22 2023-11-01T04:16:28.770 INFO:tasks.rados.rados.0.smithi190.stdout:145: oid 42 version 22 is already newer than 21 2023-11-01T04:16:28.770 INFO:tasks.rados.rados.0.smithi190.stdout:145: oid 42 updating version 22 to 23 2023-11-01T04:16:28.770 INFO:tasks.rados.rados.0.smithi190.stdout:145: oid 42 version 23 is already newer than 20 2023-11-01T04:16:28.770 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 42 v 23 (ObjNum 71 snap 8 seq_num 71) dirty exists 2023-11-01T04:16:28.771 INFO:tasks.rados.rados.0.smithi190.stdout:145: left oid 42 (ObjNum 71 snap 8 seq_num 71) 2023-11-01T04:16:28.784 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:28.785 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-01T04:16:28.803 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:28.803 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-01T04:16:28.812 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:28.810+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:28.823 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:28.823 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-01T04:16:28.843 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:28.843 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:28.844 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:28.845 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-01T04:16:28.857 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-01T04:16:28.865 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:28.865 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-01T04:16:28.882 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-01T04:16:28.903 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:28.904 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-01T04:16:28.913 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:28.911+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:28.921 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:28.921 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-01T04:16:28.955 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:28.956 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-01T04:16:28.977 INFO:teuthology.orchestra.run.smithi153.stdout:osd.12: {} 2023-11-01T04:16:28.977 INFO:teuthology.orchestra.run.smithi153.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-01T04:16:28.979 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:28.979 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:28.993 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-01T04:16:28.999 INFO:tasks.rados.rados.0.smithi190.stdout:146: finishing write tid 5 to smithi19039307-25 2023-11-01T04:16:28.999 INFO:tasks.rados.rados.0.smithi190.stdout:146: oid 25 updating version 0 to 22 2023-11-01T04:16:28.999 INFO:tasks.rados.rados.0.smithi190.stdout:146: oid 25 updating version 22 to 24 2023-11-01T04:16:29.000 INFO:tasks.rados.rados.0.smithi190.stdout:146: oid 25 version 24 is already newer than 23 2023-11-01T04:16:29.000 INFO:tasks.rados.rados.0.smithi190.stdout:146: oid 25 updating version 24 to 25 2023-11-01T04:16:29.000 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 25 v 25 (ObjNum 72 snap 8 seq_num 72) dirty exists 2023-11-01T04:16:29.000 INFO:tasks.rados.rados.0.smithi190.stdout:146: left oid 25 (ObjNum 72 snap 8 seq_num 72) 2023-11-01T04:16:29.001 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:29.001 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-01T04:16:29.014 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:29.011+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:29.021 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-01T04:16:29.044 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:29.044 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-01T04:16:29.046 INFO:teuthology.orchestra.run.smithi153.stderr:nodeep-scrub is unset 2023-11-01T04:16:29.047 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.9 to 1 (10000) 2023-11-01T04:16:29.054 INFO:tasks.rados.rados.0.smithi190.stdout:142: done (4 left) 2023-11-01T04:16:29.054 INFO:tasks.rados.rados.0.smithi190.stdout:145: done (3 left) 2023-11-01T04:16:29.055 INFO:tasks.rados.rados.0.smithi190.stdout:146: done (2 left) 2023-11-01T04:16:29.055 INFO:tasks.rados.rados.0.smithi190.stdout:147: done (1 left) 2023-11-01T04:16:29.055 INFO:tasks.rados.rados.0.smithi190.stdout:148: done (0 left) 2023-11-01T04:16:29.055 INFO:tasks.rados.rados.0.smithi190.stdout:149: read oid 33 snap -1 2023-11-01T04:16:29.055 INFO:tasks.rados.rados.0.smithi190.stdout:149: expect (ObjNum 32 snap 0 seq_num 32) 2023-11-01T04:16:29.055 INFO:tasks.rados.rados.0.smithi190.stdout:150: read oid 45 snap -1 2023-11-01T04:16:29.056 INFO:tasks.rados.rados.0.smithi190.stdout:150: expect (ObjNum 44 snap 0 seq_num 44) 2023-11-01T04:16:29.056 INFO:tasks.rados.rados.0.smithi190.stdout:151: snap_create 2023-11-01T04:16:29.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2023-11-01T04:16:29.115 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:29.114+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:29.163 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:29.163 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:29.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-01T04:16:29.216 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:29.214+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:29.302 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:29.302 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:29.316 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-01T04:16:29.317 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:29.315+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:29.365 INFO:tasks.rados.rados.0.smithi190.stdout:150: expect (ObjNum 44 snap 0 seq_num 44) 2023-11-01T04:16:29.418 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:29.415+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:29.438 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:29.438 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:29.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-01T04:16:29.467 INFO:tasks.rados.rados.0.smithi190.stdout:149: expect (ObjNum 32 snap 0 seq_num 32) 2023-11-01T04:16:29.519 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:29.518+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:29.567 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:29.567 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:29.581 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-01T04:16:29.628 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:29.627+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:29.698 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:29.698 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:29.713 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-01T04:16:29.722 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:16:29.723+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:16:29.823 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:29.821+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:29.880 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:29.881 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:29.896 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-01T04:16:29.924 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:29.923+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:30.019 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:30.019 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:30.024 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:30.023+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:30.034 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-01T04:16:30.056 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.10 to 1 (10000) 2023-11-01T04:16:30.063 INFO:tasks.rados.rados.0.smithi190.stdout:149: done (2 left) 2023-11-01T04:16:30.063 INFO:tasks.rados.rados.0.smithi190.stdout:150: done (1 left) 2023-11-01T04:16:30.063 INFO:tasks.rados.rados.0.smithi190.stdout:151: done (0 left) 2023-11-01T04:16:30.063 INFO:tasks.rados.rados.0.smithi190.stdout:152: copy_from oid 48 from oid 42 current snap is 10 2023-11-01T04:16:30.063 INFO:tasks.rados.rados.0.smithi190.stdout:153: delete oid 29 current snap is 10 2023-11-01T04:16:30.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2023-11-01T04:16:30.124 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:30.124+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:30.202 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:30.202 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:30.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-01T04:16:30.225 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:30.225+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:30.286 INFO:tasks.rados.rados.0.smithi190.stdout:152: finishing copy_from racing read to smithi19039307-48 2023-11-01T04:16:30.326 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:30.324+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:30.334 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:30.334 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:30.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-01T04:16:30.427 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:30.424+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:30.470 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:30.471 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:30.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-11-01T04:16:30.527 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:30.528+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:30.571 INFO:tasks.rados.rados.0.smithi190.stdout:153: done (1 left) 2023-11-01T04:16:30.571 INFO:tasks.rados.rados.0.smithi190.stdout:154: snap_create 2023-11-01T04:16:30.587 INFO:tasks.rados.rados.0.smithi190.stdout:152: finishing copy_from to smithi19039307-48 2023-11-01T04:16:30.587 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 48 v 23 (ObjNum 71 snap 8 seq_num 71) dirty exists 2023-11-01T04:16:30.616 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:30.617 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:30.628 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:30.625+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:30.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-11-01T04:16:30.729 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:30.725+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:30.751 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:30.752 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:30.766 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-11-01T04:16:30.830 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:30.828+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:30.887 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:30.887 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:30.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-01T04:16:30.931 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:30.928+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:31.023 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:31.023 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:31.032 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:31.031+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:31.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-01T04:16:31.060 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.11 to 1 (10000) 2023-11-01T04:16:31.067 INFO:tasks.rados.rados.0.smithi190.stdout:152: done (1 left) 2023-11-01T04:16:31.067 INFO:tasks.rados.rados.0.smithi190.stdout:154: done (0 left) 2023-11-01T04:16:31.067 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 43 current snap is 11 2023-11-01T04:16:31.067 INFO:tasks.rados.rados.0.smithi190.stdout:155: seq_num 74 ranges {0=475136,475136=622592,1097728=147456} 2023-11-01T04:16:31.069 INFO:tasks.rados.rados.0.smithi190.stdout:155: writing smithi19039307-43 from 0 to 475136 tid 1 2023-11-01T04:16:31.072 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-11-01T04:16:31.073 INFO:tasks.rados.rados.0.smithi190.stdout:155: writing smithi19039307-43 from 475136 to 1097728 tid 2 2023-11-01T04:16:31.074 INFO:tasks.rados.rados.0.smithi190.stdout:155: writing smithi19039307-43 from 1097728 to 1245184 tid 3 2023-11-01T04:16:31.074 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 10 current snap is 11 2023-11-01T04:16:31.074 INFO:tasks.rados.rados.0.smithi190.stdout:156: seq_num 75 ranges {163840=491520,655360=770048} 2023-11-01T04:16:31.077 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-01T04:16:31.078 INFO:tasks.rados.rados.0.smithi190.stdout:156: writing smithi19039307-10 from 163840 to 655360 tid 1 2023-11-01T04:16:31.082 INFO:tasks.rados.rados.0.smithi190.stdout:156: writing smithi19039307-10 from 655360 to 1425408 tid 2 2023-11-01T04:16:31.082 INFO:tasks.rados.rados.0.smithi190.stdout:157: snap_create 2023-11-01T04:16:31.086 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2023-11-01T04:16:31.162 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:31.162 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:31.171 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:31.172+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:31.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-01T04:16:31.263 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:31.263+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:31.270 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:31.270 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-11-01T04:16:31.294 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:31.295 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-01T04:16:31.314 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:31.315 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-01T04:16:31.334 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:31.334 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-01T04:16:31.334 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:31.333+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:31.343 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:31.344 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:31.349 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:31.349 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-01T04:16:31.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-01T04:16:31.368 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:31.368 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-01T04:16:31.388 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:31.388 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-01T04:16:31.406 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:31.407 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-01T04:16:31.421 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:31.422 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-01T04:16:31.435 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:31.432+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:31.441 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:31.441 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-01T04:16:31.462 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:31.462 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-01T04:16:31.481 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:31.481 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-01T04:16:31.499 INFO:teuthology.orchestra.run.smithi153.stdout:osd.12: {} 2023-11-01T04:16:31.500 INFO:teuthology.orchestra.run.smithi153.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-01T04:16:31.521 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:31.521 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:31.521 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2023-11-01T04:16:31.522 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:31.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-01T04:16:31.536 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:31.533+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:31.581 INFO:tasks.rados.rados.0.smithi190.stdout:155: finishing write tid 1 to smithi19039307-43 2023-11-01T04:16:31.595 INFO:tasks.rados.rados.0.smithi190.stdout:156: finishing write tid 1 to smithi19039307-10 2023-11-01T04:16:31.596 INFO:tasks.rados.rados.0.smithi190.stdout:156: finishing write tid 2 to smithi19039307-10 2023-11-01T04:16:31.596 INFO:tasks.rados.rados.0.smithi190.stdout:156: finishing write tid 3 to smithi19039307-10 2023-11-01T04:16:31.637 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:31.635+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:31.656 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:31.656 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:31.657 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2023-11-01T04:16:31.657 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2023-11-01T04:16:31.657 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:31.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-01T04:16:31.681 INFO:tasks.rados.rados.0.smithi190.stdout:155: finishing write tid 2 to smithi19039307-43 2023-11-01T04:16:31.682 INFO:tasks.rados.rados.0.smithi190.stdout:155: finishing write tid 3 to smithi19039307-43 2023-11-01T04:16:31.703 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:31.704 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-01T04:16:31.729 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:31.729 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-01T04:16:31.738 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:31.736+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:31.751 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:31.752 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-01T04:16:31.782 INFO:tasks.rados.rados.0.smithi190.stdout:155: finishing write tid 4 to smithi19039307-43 2023-11-01T04:16:31.800 INFO:tasks.rados.rados.0.smithi190.stdout:156: finishing write tid 4 to smithi19039307-10 2023-11-01T04:16:31.801 INFO:tasks.rados.rados.0.smithi190.stdout:156: oid 10 updating version 0 to 17 2023-11-01T04:16:31.801 INFO:tasks.rados.rados.0.smithi190.stdout:156: oid 10 updating version 17 to 18 2023-11-01T04:16:31.801 INFO:tasks.rados.rados.0.smithi190.stdout:156: oid 10 updating version 18 to 19 2023-11-01T04:16:31.801 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 10 v 19 (ObjNum 75 snap 11 seq_num 75) dirty exists 2023-11-01T04:16:31.801 INFO:tasks.rados.rados.0.smithi190.stdout:156: left oid 10 (ObjNum 75 snap 11 seq_num 75) 2023-11-01T04:16:31.806 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:31.807 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2023-11-01T04:16:31.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2023-11-01T04:16:31.807 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2023-11-01T04:16:31.807 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.807 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:185 3.es0 3:723efae4:::smithi19039307-36:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:31.807 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.963781+0000", 2023-11-01T04:16:31.808 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.841877350000001, 2023-11-01T04:16:31.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31012846799999999, 2023-11-01T04:16:31.808 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.808 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:31.808 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:31.809 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:31.809 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:31.809 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185 2023-11-01T04:16:31.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.809 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963781+0000", 2023-11-01T04:16:31.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963781+0000", 2023-11-01T04:16:31.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963785+0000", 2023-11-01T04:16:31.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.563e-06 2023-11-01T04:16:31.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963797+0000", 2023-11-01T04:16:31.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2201999999999999e-05 2023-11-01T04:16:31.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963839+0000", 2023-11-01T04:16:31.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2181000000000002e-05 2023-11-01T04:16:31.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963864+0000", 2023-11-01T04:16:31.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4419000000000001e-05 2023-11-01T04:16:31.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964036+0000", 2023-11-01T04:16:31.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017209500000000001 2023-11-01T04:16:31.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:31.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964291+0000", 2023-11-01T04:16:31.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025550699999999998 2023-11-01T04:16:31.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:31.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966403+0000", 2023-11-01T04:16:31.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021113500000000001 2023-11-01T04:16:31.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:31.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.072479+0000", 2023-11-01T04:16:31.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10607620800000001 2023-11-01T04:16:31.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:31.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.273808+0000", 2023-11-01T04:16:31.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20132876299999999 2023-11-01T04:16:31.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:31.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.273910+0000", 2023-11-01T04:16:31.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010218 2023-11-01T04:16:31.820 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.820 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:31.820 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.821 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:186 3.es0 3:723efae4:::smithi19039307-36:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:31.821 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.963910+0000", 2023-11-01T04:16:31.821 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.841748805, 2023-11-01T04:16:31.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.410473274, 2023-11-01T04:16:31.822 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.822 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:31.822 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:31.822 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:31.822 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:31.823 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 186 2023-11-01T04:16:31.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.823 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963910+0000", 2023-11-01T04:16:31.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963910+0000", 2023-11-01T04:16:31.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963911+0000", 2023-11-01T04:16:31.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.28e-06 2023-11-01T04:16:31.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963917+0000", 2023-11-01T04:16:31.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1829999999999999e-06 2023-11-01T04:16:31.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963942+0000", 2023-11-01T04:16:31.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4808e-05 2023-11-01T04:16:31.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963957+0000", 2023-11-01T04:16:31.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4698999999999999e-05 2023-11-01T04:16:31.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.967838+0000", 2023-11-01T04:16:31.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003881139 2023-11-01T04:16:31.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:31.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.968035+0000", 2023-11-01T04:16:31.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019704400000000001 2023-11-01T04:16:31.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:31.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.968889+0000", 2023-11-01T04:16:31.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00085385099999999998 2023-11-01T04:16:31.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:31.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.072489+0000", 2023-11-01T04:16:31.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.103600028 2023-11-01T04:16:31.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:31.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374278+0000", 2023-11-01T04:16:31.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.301788789 2023-11-01T04:16:31.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:31.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374383+0000", 2023-11-01T04:16:31.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010545299999999999 2023-11-01T04:16:31.834 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.834 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:31.834 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.835 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:187 3.es0 3:723efae4:::smithi19039307-36:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:31.835 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.964019+0000", 2023-11-01T04:16:31.835 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.841640042, 2023-11-01T04:16:31.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41093530900000003, 2023-11-01T04:16:31.835 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.835 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:31.836 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:31.836 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:31.836 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:31.836 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 187 2023-11-01T04:16:31.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.836 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.837 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964019+0000", 2023-11-01T04:16:31.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964019+0000", 2023-11-01T04:16:31.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964020+0000", 2023-11-01T04:16:31.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.26e-06 2023-11-01T04:16:31.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964026+0000", 2023-11-01T04:16:31.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3260000000000001e-06 2023-11-01T04:16:31.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964049+0000", 2023-11-01T04:16:31.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2255e-05 2023-11-01T04:16:31.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.841 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.841 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.841 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964067+0000", 2023-11-01T04:16:31.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8643000000000001e-05 2023-11-01T04:16:31.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.969573+0000", 2023-11-01T04:16:31.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055055729999999997 2023-11-01T04:16:31.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.843 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:31.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.969770+0000", 2023-11-01T04:16:31.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000196787 2023-11-01T04:16:31.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.843 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.843 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:31.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.971048+0000", 2023-11-01T04:16:31.844 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012779709999999999 2023-11-01T04:16:31.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.844 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:31.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.072491+0000", 2023-11-01T04:16:31.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101443751 2023-11-01T04:16:31.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.845 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.845 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:31.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374869+0000", 2023-11-01T04:16:31.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.302377644 2023-11-01T04:16:31.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.846 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:31.846 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374954+0000", 2023-11-01T04:16:31.846 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5099000000000006e-05 2023-11-01T04:16:31.846 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.847 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:31.847 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.847 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:194 3.es0 3:706dddd5:::smithi19039307-14:head [copy-from ver 6] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:31.848 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966030+0000", 2023-11-01T04:16:31.848 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.839628509000001, 2023-11-01T04:16:31.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.72279825099999995, 2023-11-01T04:16:31.848 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.848 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:31.848 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:31.849 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:31.849 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:31.849 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 194 2023-11-01T04:16:31.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.849 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.850 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966030+0000", 2023-11-01T04:16:31.850 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.850 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966030+0000", 2023-11-01T04:16:31.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966032+0000", 2023-11-01T04:16:31.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9470000000000002e-06 2023-11-01T04:16:31.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.852 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966043+0000", 2023-11-01T04:16:31.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0396e-05 2023-11-01T04:16:31.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966073+0000", 2023-11-01T04:16:31.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0518000000000002e-05 2023-11-01T04:16:31.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966093+0000", 2023-11-01T04:16:31.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9955000000000001e-05 2023-11-01T04:16:31.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.071960+0000", 2023-11-01T04:16:31.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.105867171 2023-11-01T04:16:31.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:31.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.072114+0000", 2023-11-01T04:16:31.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000153323 2023-11-01T04:16:31.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:31.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.480839+0000", 2023-11-01T04:16:31.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40872525500000001 2023-11-01T04:16:31.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.857 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:31.857 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.487356+0000", 2023-11-01T04:16:31.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0065175770000000001 2023-11-01T04:16:31.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:31.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.688636+0000", 2023-11-01T04:16:31.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20127930899999999 2023-11-01T04:16:31.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:31.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.688829+0000", 2023-11-01T04:16:31.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0001928 2023-11-01T04:16:31.859 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.860 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:31.860 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.860 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:195 3.es0 3:706dddd5:::smithi19039307-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:31.860 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966129+0000", 2023-11-01T04:16:31.861 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.839530219, 2023-11-01T04:16:31.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51772273199999996, 2023-11-01T04:16:31.861 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.861 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:31.861 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:31.861 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:31.862 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:31.862 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 195 2023-11-01T04:16:31.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.862 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966129+0000", 2023-11-01T04:16:31.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966129+0000", 2023-11-01T04:16:31.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966130+0000", 2023-11-01T04:16:31.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4529999999999999e-06 2023-11-01T04:16:31.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966168+0000", 2023-11-01T04:16:31.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7792e-05 2023-11-01T04:16:31.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966192+0000", 2023-11-01T04:16:31.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4083999999999998e-05 2023-11-01T04:16:31.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966210+0000", 2023-11-01T04:16:31.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8196e-05 2023-11-01T04:16:31.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.173008+0000", 2023-11-01T04:16:31.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20679805200000001 2023-11-01T04:16:31.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2023-11-01T04:16:31.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.173168+0000", 2023-11-01T04:16:31.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000159457 2023-11-01T04:16:31.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.483564+0000", 2023-11-01T04:16:31.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31039652000000001 2023-11-01T04:16:31.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:31.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.483711+0000", 2023-11-01T04:16:31.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014698099999999999 2023-11-01T04:16:31.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:31.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.483851+0000", 2023-11-01T04:16:31.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000140197 2023-11-01T04:16:31.872 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.872 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:31.872 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.873 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:202 3.es0 3:7f8a06cd:::smithi19039307-49:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:31.873 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966420+0000", 2023-11-01T04:16:31.873 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.839238911000001, 2023-11-01T04:16:31.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.91933000499999995, 2023-11-01T04:16:31.873 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.874 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:31.874 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:31.874 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:31.874 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:31.874 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 202 2023-11-01T04:16:31.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.875 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966420+0000", 2023-11-01T04:16:31.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966420+0000", 2023-11-01T04:16:31.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966421+0000", 2023-11-01T04:16:31.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6199999999999999e-06 2023-11-01T04:16:31.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966431+0000", 2023-11-01T04:16:31.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4660000000000005e-06 2023-11-01T04:16:31.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966456+0000", 2023-11-01T04:16:31.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4748000000000001e-05 2023-11-01T04:16:31.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966473+0000", 2023-11-01T04:16:31.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7323e-05 2023-11-01T04:16:31.880 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.880 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.880 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.880 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.375871+0000", 2023-11-01T04:16:31.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40939844800000003 2023-11-01T04:16:31.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.881 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:31.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.376008+0000", 2023-11-01T04:16:31.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000136997 2023-11-01T04:16:31.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.882 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:31.882 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.585026+0000", 2023-11-01T04:16:31.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.209017436 2023-11-01T04:16:31.882 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:31.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.688063+0000", 2023-11-01T04:16:31.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.103037031 2023-11-01T04:16:31.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:31.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.885528+0000", 2023-11-01T04:16:31.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.19746507999999999 2023-11-01T04:16:31.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:31.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.885750+0000", 2023-11-01T04:16:31.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022185600000000001 2023-11-01T04:16:31.885 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.885 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:31.885 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.886 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:203 3.es0 3:7f8a06cd:::smithi19039307-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:31.886 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966500+0000", 2023-11-01T04:16:31.886 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.839158512999999, 2023-11-01T04:16:31.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.62108323700000001, 2023-11-01T04:16:31.886 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.887 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:31.887 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:31.887 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:31.887 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:31.887 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 203 2023-11-01T04:16:31.887 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.888 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966500+0000", 2023-11-01T04:16:31.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.889 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.889 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966500+0000", 2023-11-01T04:16:31.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966501+0000", 2023-11-01T04:16:31.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1570000000000001e-06 2023-11-01T04:16:31.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966530+0000", 2023-11-01T04:16:31.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8337999999999999e-05 2023-11-01T04:16:31.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966549+0000", 2023-11-01T04:16:31.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9624999999999999e-05 2023-11-01T04:16:31.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966584+0000", 2023-11-01T04:16:31.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4442e-05 2023-11-01T04:16:31.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.376334+0000", 2023-11-01T04:16:31.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.409750269 2023-11-01T04:16:31.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.894 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.894 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2023-11-01T04:16:31.894 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.376468+0000", 2023-11-01T04:16:31.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000133588 2023-11-01T04:16:31.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.895 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.895 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.586951+0000", 2023-11-01T04:16:31.895 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21048317499999999 2023-11-01T04:16:31.895 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.896 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:31.896 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.587248+0000", 2023-11-01T04:16:31.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029743400000000001 2023-11-01T04:16:31.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.896 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.896 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:31.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.587583+0000", 2023-11-01T04:16:31.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000335209 2023-11-01T04:16:31.897 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.897 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:31.897 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.898 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=22, last_complete=37'15, committed=1, applied=1))", 2023-11-01T04:16:31.898 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.073048+0000", 2023-11-01T04:16:31.898 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.732610605, 2023-11-01T04:16:31.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30149571800000002, 2023-11-01T04:16:31.898 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.899 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:31.899 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073048+0000", 2023-11-01T04:16:31.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.900 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073048+0000", 2023-11-01T04:16:31.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073049+0000", 2023-11-01T04:16:31.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.96e-07 2023-11-01T04:16:31.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.902 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.902 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073052+0000", 2023-11-01T04:16:31.902 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9739999999999998e-06 2023-11-01T04:16:31.902 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.903 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.903 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073061+0000", 2023-11-01T04:16:31.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.003e-06 2023-11-01T04:16:31.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.904 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073067+0000", 2023-11-01T04:16:31.904 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6560000000000003e-06 2023-11-01T04:16:31.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374508+0000", 2023-11-01T04:16:31.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30144055400000003 2023-11-01T04:16:31.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.905 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:31.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374544+0000", 2023-11-01T04:16:31.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6134999999999999e-05 2023-11-01T04:16:31.906 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.906 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:31.906 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.907 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=21, last_complete=37'14, committed=1, applied=1))", 2023-11-01T04:16:31.907 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.073145+0000", 2023-11-01T04:16:31.907 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.732513390999999, 2023-11-01T04:16:31.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30125827300000002, 2023-11-01T04:16:31.907 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.907 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:31.908 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.908 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.908 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.908 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073145+0000", 2023-11-01T04:16:31.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.908 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073145+0000", 2023-11-01T04:16:31.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073146+0000", 2023-11-01T04:16:31.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8599999999999999e-07 2023-11-01T04:16:31.910 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.911 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073149+0000", 2023-11-01T04:16:31.911 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8339999999999999e-06 2023-11-01T04:16:31.911 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073160+0000", 2023-11-01T04:16:31.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1671999999999999e-05 2023-11-01T04:16:31.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.913 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073168+0000", 2023-11-01T04:16:31.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2269999999999996e-06 2023-11-01T04:16:31.913 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.913 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.913 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374161+0000", 2023-11-01T04:16:31.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30099383499999999 2023-11-01T04:16:31.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:31.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374404+0000", 2023-11-01T04:16:31.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024231900000000001 2023-11-01T04:16:31.915 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.915 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:31.915 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.916 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=22, last_complete=37'15, committed=1, applied=1))", 2023-11-01T04:16:31.916 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.073177+0000", 2023-11-01T04:16:31.916 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.732481460000001, 2023-11-01T04:16:31.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30259569400000003, 2023-11-01T04:16:31.916 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.917 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:31.917 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073177+0000", 2023-11-01T04:16:31.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073177+0000", 2023-11-01T04:16:31.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.919 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.919 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073178+0000", 2023-11-01T04:16:31.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6300000000000001e-07 2023-11-01T04:16:31.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.920 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073180+0000", 2023-11-01T04:16:31.920 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.65e-06 2023-11-01T04:16:31.921 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.921 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.921 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.921 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073188+0000", 2023-11-01T04:16:31.921 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0730000000000005e-06 2023-11-01T04:16:31.921 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.922 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073194+0000", 2023-11-01T04:16:31.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1800000000000001e-06 2023-11-01T04:16:31.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374769+0000", 2023-11-01T04:16:31.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30157417600000003 2023-11-01T04:16:31.923 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:31.924 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.375773+0000", 2023-11-01T04:16:31.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001004352 2023-11-01T04:16:31.924 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.924 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:31.925 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.925 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=22, last_complete=37'15, committed=1, applied=1))", 2023-11-01T04:16:31.925 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.075657+0000", 2023-11-01T04:16:31.925 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.730001551000001, 2023-11-01T04:16:31.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29898018399999998, 2023-11-01T04:16:31.926 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.926 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:31.926 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.075657+0000", 2023-11-01T04:16:31.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.075657+0000", 2023-11-01T04:16:31.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.075659+0000", 2023-11-01T04:16:31.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5829999999999999e-06 2023-11-01T04:16:31.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.075668+0000", 2023-11-01T04:16:31.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9889999999999998e-06 2023-11-01T04:16:31.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.075696+0000", 2023-11-01T04:16:31.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8218e-05 2023-11-01T04:16:31.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.931 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.931 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.075712+0000", 2023-11-01T04:16:31.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5736e-05 2023-11-01T04:16:31.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374611+0000", 2023-11-01T04:16:31.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29889970700000001 2023-11-01T04:16:31.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:31.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374637+0000", 2023-11-01T04:16:31.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5950999999999999e-05 2023-11-01T04:16:31.933 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.933 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:31.934 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.934 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=24, last_complete=37'17, committed=1, applied=1))", 2023-11-01T04:16:31.934 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.484201+0000", 2023-11-01T04:16:31.934 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.321457669000001, 2023-11-01T04:16:31.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20359449700000001, 2023-11-01T04:16:31.935 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.935 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:31.935 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.936 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.484201+0000", 2023-11-01T04:16:31.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.936 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.484201+0000", 2023-11-01T04:16:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.484209+0000", 2023-11-01T04:16:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8229999999999995e-06 2023-11-01T04:16:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.484220+0000", 2023-11-01T04:16:31.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1039999999999999e-05 2023-11-01T04:16:31.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.484245+0000", 2023-11-01T04:16:31.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5298000000000001e-05 2023-11-01T04:16:31.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.940 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.584332+0000", 2023-11-01T04:16:31.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100086439 2023-11-01T04:16:31.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.941 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.941 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.687740+0000", 2023-11-01T04:16:31.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.103407998 2023-11-01T04:16:31.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.942 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:31.942 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.687796+0000", 2023-11-01T04:16:31.942 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5899000000000001e-05 2023-11-01T04:16:31.942 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.942 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:31.943 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.943 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:270 3.es0 3:706dddd5:::smithi19039307-14:head [read 0~638976,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-01T04:16:31.943 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.017359+0000", 2023-11-01T04:16:31.943 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.788300096, 2023-11-01T04:16:31.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21120662700000001, 2023-11-01T04:16:31.944 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.944 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:31.944 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:31.944 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:31.944 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:31.945 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 270 2023-11-01T04:16:31.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.945 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.946 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017359+0000", 2023-11-01T04:16:31.946 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.946 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.946 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.946 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017359+0000", 2023-11-01T04:16:31.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.947 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.947 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.947 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017363+0000", 2023-11-01T04:16:31.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6800000000000001e-06 2023-11-01T04:16:31.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.948 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017382+0000", 2023-11-01T04:16:31.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8495000000000002e-05 2023-11-01T04:16:31.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.949 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017463+0000", 2023-11-01T04:16:31.949 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1058999999999998e-05 2023-11-01T04:16:31.950 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017483+0000", 2023-11-01T04:16:31.950 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9718999999999999e-05 2023-11-01T04:16:31.950 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.951 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.951 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.951 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017613+0000", 2023-11-01T04:16:31.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000130109 2023-11-01T04:16:31.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.951 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.952 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2023-11-01T04:16:31.952 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017624+0000", 2023-11-01T04:16:31.952 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1389999999999999e-05 2023-11-01T04:16:31.952 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.952 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.953 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.021914+0000", 2023-11-01T04:16:31.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042898889999999999 2023-11-01T04:16:31.953 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:31.954 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.022163+0000", 2023-11-01T04:16:31.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024884899999999999 2023-11-01T04:16:31.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.954 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:31.954 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.228565+0000", 2023-11-01T04:16:31.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20640243699999999 2023-11-01T04:16:31.955 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.955 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:31.955 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.956 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:271 3.es0 3:706dddd5:::smithi19039307-14:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-01T04:16:31.956 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.017519+0000", 2023-11-01T04:16:31.956 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.788139648, 2023-11-01T04:16:31.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31206822200000001, 2023-11-01T04:16:31.956 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.956 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:31.957 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:31.957 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:31.957 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:31.957 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 271 2023-11-01T04:16:31.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.958 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017519+0000", 2023-11-01T04:16:31.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017519+0000", 2023-11-01T04:16:31.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.960 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.960 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017520+0000", 2023-11-01T04:16:31.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.203e-06 2023-11-01T04:16:31.960 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017527+0000", 2023-11-01T04:16:31.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7089999999999996e-06 2023-11-01T04:16:31.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017559+0000", 2023-11-01T04:16:31.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1562e-05 2023-11-01T04:16:31.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017568+0000", 2023-11-01T04:16:31.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7189999999999996e-06 2023-11-01T04:16:31.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017754+0000", 2023-11-01T04:16:31.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018541 2023-11-01T04:16:31.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:31.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017764+0000", 2023-11-01T04:16:31.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0609e-05 2023-11-01T04:16:31.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.023462+0000", 2023-11-01T04:16:31.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0056980370000000004 2023-11-01T04:16:31.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:31.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.023706+0000", 2023-11-01T04:16:31.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024344300000000001 2023-11-01T04:16:31.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:31.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.329587+0000", 2023-11-01T04:16:31.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30588153000000001 2023-11-01T04:16:31.968 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.968 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:31.968 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.968 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:272 3.es0 3:706dddd5:::smithi19039307-14:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-01T04:16:31.969 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.017583+0000", 2023-11-01T04:16:31.969 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.788075992, 2023-11-01T04:16:31.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.412902926, 2023-11-01T04:16:31.969 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.969 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:31.970 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:31.970 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:31.970 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:31.970 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 272 2023-11-01T04:16:31.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.970 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017583+0000", 2023-11-01T04:16:31.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017583+0000", 2023-11-01T04:16:31.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017584+0000", 2023-11-01T04:16:31.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6000000000000002e-07 2023-11-01T04:16:31.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017588+0000", 2023-11-01T04:16:31.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1330000000000001e-06 2023-11-01T04:16:31.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017598+0000", 2023-11-01T04:16:31.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8430000000000005e-06 2023-11-01T04:16:31.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017605+0000", 2023-11-01T04:16:31.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7030000000000006e-06 2023-11-01T04:16:31.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017822+0000", 2023-11-01T04:16:31.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000217137 2023-11-01T04:16:31.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:31.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017832+0000", 2023-11-01T04:16:31.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2459999999999992e-06 2023-11-01T04:16:31.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.226201+0000", 2023-11-01T04:16:31.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20836893100000001 2023-11-01T04:16:31.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:31.981 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.226369+0000", 2023-11-01T04:16:31.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016842100000000001 2023-11-01T04:16:31.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.981 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:31.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.430486+0000", 2023-11-01T04:16:31.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20411665200000001 2023-11-01T04:16:31.982 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.982 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:31.982 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.983 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 41/31 ECSubWriteReply(tid=39, last_complete=41'29, committed=1, applied=1))", 2023-11-01T04:16:31.983 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.852996+0000", 2023-11-01T04:16:31.983 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.952663007, 2023-11-01T04:16:31.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30060843599999998, 2023-11-01T04:16:31.984 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.984 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:31.984 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.852996+0000", 2023-11-01T04:16:31.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.985 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.852996+0000", 2023-11-01T04:16:31.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.852997+0000", 2023-11-01T04:16:31.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6599999999999994e-07 2023-11-01T04:16:31.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853007+0000", 2023-11-01T04:16:31.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9429999999999995e-06 2023-11-01T04:16:31.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853024+0000", 2023-11-01T04:16:31.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6889e-05 2023-11-01T04:16:31.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.989 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.989 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853034+0000", 2023-11-01T04:16:31.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0665999999999999e-05 2023-11-01T04:16:31.990 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.990 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.990 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:22.153586+0000", 2023-11-01T04:16:31.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30055170399999998 2023-11-01T04:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:22.153604+0000", 2023-11-01T04:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8267999999999999e-05 2023-11-01T04:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 41/31 ECSubWriteReply(tid=39, last_complete=41'29, committed=1, applied=1))", 2023-11-01T04:16:31.993 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.853045+0000", 2023-11-01T04:16:31.993 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.9526141149999994, 2023-11-01T04:16:31.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30061494799999999, 2023-11-01T04:16:31.993 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:31.993 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:31.993 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:31.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:31.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853045+0000", 2023-11-01T04:16:31.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:31.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853045+0000", 2023-11-01T04:16:31.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:31.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:31.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853046+0000", 2023-11-01T04:16:31.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8700000000000004e-07 2023-11-01T04:16:31.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:31.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853049+0000", 2023-11-01T04:16:31.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.54e-06 2023-11-01T04:16:31.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:31.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853058+0000", 2023-11-01T04:16:31.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1540000000000008e-06 2023-11-01T04:16:31.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:31.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853064+0000", 2023-11-01T04:16:31.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6459999999999998e-06 2023-11-01T04:16:31.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:31.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:31.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:31.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:22.153623+0000", 2023-11-01T04:16:31.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30055869000000002 2023-11-01T04:16:32.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:22.153660+0000", 2023-11-01T04:16:32.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7030999999999998e-05 2023-11-01T04:16:32.000 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.001 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.001 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.001 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.8s2 46/31 ECSubWrite(tid=27, reqid=client.4559.0:327, at_version=46'18, trim_to=0'0, roll_forward_to=43'14))", 2023-11-01T04:16:32.001 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:26.186950+0000", 2023-11-01T04:16:32.002 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.6187082750000004, 2023-11-01T04:16:32.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.204465387, 2023-11-01T04:16:32.002 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.002 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:32.002 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.186950+0000", 2023-11-01T04:16:32.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.186950+0000", 2023-11-01T04:16:32.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.186952+0000", 2023-11-01T04:16:32.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6959999999999999e-06 2023-11-01T04:16:32.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.186996+0000", 2023-11-01T04:16:32.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3847999999999998e-05 2023-11-01T04:16:32.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.187035+0000", 2023-11-01T04:16:32.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8631999999999999e-05 2023-11-01T04:16:32.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.287127+0000", 2023-11-01T04:16:32.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.1000924 2023-11-01T04:16:32.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.387388+0000", 2023-11-01T04:16:32.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100260951 2023-11-01T04:16:32.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:32.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.387404+0000", 2023-11-01T04:16:32.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6099999999999998e-05 2023-11-01T04:16:32.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:32.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.391320+0000", 2023-11-01T04:16:32.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003916302 2023-11-01T04:16:32.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.391416+0000", 2023-11-01T04:16:32.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5458000000000007e-05 2023-11-01T04:16:32.012 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.012 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.012 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.012 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.012 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:350 3.es0 3:706dddd5:::smithi19039307-14:head [delete] snapc c=[c,b,4] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:32.013 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:27.032803+0000", 2023-11-01T04:16:32.013 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 4.7728554140000004, 2023-11-01T04:16:32.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.213835843, 2023-11-01T04:16:32.013 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.013 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.013 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:32.014 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:32.014 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:32.014 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 350 2023-11-01T04:16:32.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.014 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:27.032803+0000", 2023-11-01T04:16:32.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:27.032803+0000", 2023-11-01T04:16:32.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:27.032807+0000", 2023-11-01T04:16:32.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5769999999999998e-06 2023-11-01T04:16:32.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:27.032820+0000", 2023-11-01T04:16:32.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3216e-05 2023-11-01T04:16:32.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:27.032861+0000", 2023-11-01T04:16:32.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0465e-05 2023-11-01T04:16:32.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:27.032877+0000", 2023-11-01T04:16:32.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6322000000000001e-05 2023-11-01T04:16:32.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:27.032950+0000", 2023-11-01T04:16:32.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3060999999999997e-05 2023-11-01T04:16:32.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2023-11-01T04:16:32.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:27.032955+0000", 2023-11-01T04:16:32.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4349999999999996e-06 2023-11-01T04:16:32.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:27.036517+0000", 2023-11-01T04:16:32.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0035615199999999999 2023-11-01T04:16:32.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:27.036698+0000", 2023-11-01T04:16:32.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000181091 2023-11-01T04:16:32.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:32.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:27.038321+0000", 2023-11-01T04:16:32.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016225569999999999 2023-11-01T04:16:32.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:32.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:27.144067+0000", 2023-11-01T04:16:32.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10574620699999999 2023-11-01T04:16:32.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:27.246518+0000", 2023-11-01T04:16:32.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10245164399999999 2023-11-01T04:16:32.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:27.246639+0000", 2023-11-01T04:16:32.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120748 2023-11-01T04:16:32.027 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.027 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.028 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.028 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.7s1 48/31 ECSubWrite(tid=45, reqid=client.4559.0:372, at_version=48'22, trim_to=0'0, roll_forward_to=46'20))", 2023-11-01T04:16:32.028 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:28.185976+0000", 2023-11-01T04:16:32.029 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 3.6196824460000001, 2023-11-01T04:16:32.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20381165500000001, 2023-11-01T04:16:32.029 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.029 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:32.029 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:28.185976+0000", 2023-11-01T04:16:32.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:28.185976+0000", 2023-11-01T04:16:32.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:28.185977+0000", 2023-11-01T04:16:32.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1699999999999998e-07 2023-11-01T04:16:32.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:28.186166+0000", 2023-11-01T04:16:32.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000189171 2023-11-01T04:16:32.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:28.186200+0000", 2023-11-01T04:16:32.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4115000000000001e-05 2023-11-01T04:16:32.034 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.034 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.034 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.034 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:28.186227+0000", 2023-11-01T04:16:32.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6862000000000001e-05 2023-11-01T04:16:32.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:28.286493+0000", 2023-11-01T04:16:32.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10026587000000001 2023-11-01T04:16:32.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:32.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:28.286531+0000", 2023-11-01T04:16:32.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8204000000000002e-05 2023-11-01T04:16:32.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:32.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:28.389730+0000", 2023-11-01T04:16:32.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.103198529 2023-11-01T04:16:32.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:28.389788+0000", 2023-11-01T04:16:32.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8286999999999999e-05 2023-11-01T04:16:32.038 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.038 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.038 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.039 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.039 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.039 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:32.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-01T04:16:32.041 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:32.038+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:32.079 INFO:teuthology.orchestra.run.smithi153.stderr:noscrub is set 2023-11-01T04:16:32.079 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.12 to 1 (10000) 2023-11-01T04:16:32.086 INFO:tasks.rados.rados.0.smithi190.stdout:155: finishing write tid 5 to smithi19039307-43 2023-11-01T04:16:32.086 INFO:tasks.rados.rados.0.smithi190.stdout:155: oid 43 updating version 0 to 34 2023-11-01T04:16:32.086 INFO:tasks.rados.rados.0.smithi190.stdout:155: oid 43 updating version 34 to 35 2023-11-01T04:16:32.086 INFO:tasks.rados.rados.0.smithi190.stdout:155: oid 43 updating version 35 to 37 2023-11-01T04:16:32.087 INFO:tasks.rados.rados.0.smithi190.stdout:155: oid 43 version 37 is already newer than 36 2023-11-01T04:16:32.087 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 43 v 37 (ObjNum 74 snap 11 seq_num 74) dirty exists 2023-11-01T04:16:32.087 INFO:tasks.rados.rados.0.smithi190.stdout:155: left oid 43 (ObjNum 74 snap 11 seq_num 74) 2023-11-01T04:16:32.089 INFO:tasks.rados.rados.0.smithi190.stdout:155: done (2 left) 2023-11-01T04:16:32.090 INFO:tasks.rados.rados.0.smithi190.stdout:156: done (1 left) 2023-11-01T04:16:32.090 INFO:tasks.rados.rados.0.smithi190.stdout:157: done (0 left) 2023-11-01T04:16:32.090 INFO:tasks.rados.rados.0.smithi190.stdout:158: snap_create 2023-11-01T04:16:32.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2023-11-01T04:16:32.141 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:32.138+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:32.210 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:32.211 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:32.211 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2023-11-01T04:16:32.211 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:32.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-01T04:16:32.268 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:32.266+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:32.343 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:32.344+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:32.391 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:32.391 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:32.391 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2023-11-01T04:16:32.391 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2023-11-01T04:16:32.392 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:32.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-01T04:16:32.444 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:32.444+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:32.526 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:32.527 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2023-11-01T04:16:32.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2023-11-01T04:16:32.527 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2023-11-01T04:16:32.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.528 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:36 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'22, mlcod=25'16)", 2023-11-01T04:16:32.528 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:00.688757+0000", 2023-11-01T04:16:32.528 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.838348202999999, 2023-11-01T04:16:32.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20189465300000001, 2023-11-01T04:16:32.528 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.529 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.529 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688757+0000", 2023-11-01T04:16:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688757+0000", 2023-11-01T04:16:32.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688758+0000", 2023-11-01T04:16:32.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5300000000000004e-07 2023-11-01T04:16:32.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688763+0000", 2023-11-01T04:16:32.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7389999999999996e-06 2023-11-01T04:16:32.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688791+0000", 2023-11-01T04:16:32.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8136999999999998e-05 2023-11-01T04:16:32.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688799+0000", 2023-11-01T04:16:32.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3390000000000004e-06 2023-11-01T04:16:32.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.535 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.888843+0000", 2023-11-01T04:16:32.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20004432999999999 2023-11-01T04:16:32.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.888878+0000", 2023-11-01T04:16:32.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4814000000000002e-05 2023-11-01T04:16:32.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.537 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.890502+0000", 2023-11-01T04:16:32.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001623813 2023-11-01T04:16:32.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.890652+0000", 2023-11-01T04:16:32.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014982799999999999 2023-11-01T04:16:32.539 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.539 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.540 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.540 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:37 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'23, mlcod=25'16)", 2023-11-01T04:16:32.540 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:00.688817+0000", 2023-11-01T04:16:32.541 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.838288403, 2023-11-01T04:16:32.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.202682319, 2023-11-01T04:16:32.541 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.541 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.541 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688817+0000", 2023-11-01T04:16:32.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688817+0000", 2023-11-01T04:16:32.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688817+0000", 2023-11-01T04:16:32.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6700000000000003e-07 2023-11-01T04:16:32.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688823+0000", 2023-11-01T04:16:32.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1589999999999998e-06 2023-11-01T04:16:32.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.546 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688835+0000", 2023-11-01T04:16:32.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1472e-05 2023-11-01T04:16:32.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.788922+0000", 2023-11-01T04:16:32.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100087538 2023-11-01T04:16:32.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.889458+0000", 2023-11-01T04:16:32.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100535469 2023-11-01T04:16:32.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.549 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.889515+0000", 2023-11-01T04:16:32.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6990999999999997e-05 2023-11-01T04:16:32.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.891416+0000", 2023-11-01T04:16:32.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019012409999999999 2023-11-01T04:16:32.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.891499+0000", 2023-11-01T04:16:32.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2881999999999997e-05 2023-11-01T04:16:32.551 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.551 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.552 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.552 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:53 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'31, mlcod=26'29)", 2023-11-01T04:16:32.552 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:01.627435+0000", 2023-11-01T04:16:32.553 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.899670104999998, 2023-11-01T04:16:32.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20423751000000001, 2023-11-01T04:16:32.553 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.553 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.553 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627435+0000", 2023-11-01T04:16:32.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.555 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.555 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627435+0000", 2023-11-01T04:16:32.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627437+0000", 2023-11-01T04:16:32.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6670000000000001e-06 2023-11-01T04:16:32.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627449+0000", 2023-11-01T04:16:32.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2571999999999999e-05 2023-11-01T04:16:32.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627477+0000", 2023-11-01T04:16:32.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.741e-05 2023-11-01T04:16:32.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627491+0000", 2023-11-01T04:16:32.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3953e-05 2023-11-01T04:16:32.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627659+0000", 2023-11-01T04:16:32.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016794299999999999 2023-11-01T04:16:32.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627699+0000", 2023-11-01T04:16:32.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0036000000000001e-05 2023-11-01T04:16:32.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831530+0000", 2023-11-01T04:16:32.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20383152299999999 2023-11-01T04:16:32.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831672+0000", 2023-11-01T04:16:32.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014240600000000001 2023-11-01T04:16:32.563 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.563 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.563 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.564 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:54 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'32, mlcod=26'29)", 2023-11-01T04:16:32.564 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:01.628360+0000", 2023-11-01T04:16:32.564 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.898745326, 2023-11-01T04:16:32.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20340096899999999, 2023-11-01T04:16:32.564 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.565 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.565 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628360+0000", 2023-11-01T04:16:32.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628360+0000", 2023-11-01T04:16:32.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.567 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628362+0000", 2023-11-01T04:16:32.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.787e-06 2023-11-01T04:16:32.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628374+0000", 2023-11-01T04:16:32.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2499e-05 2023-11-01T04:16:32.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628399+0000", 2023-11-01T04:16:32.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5037000000000001e-05 2023-11-01T04:16:32.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628412+0000", 2023-11-01T04:16:32.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2982e-05 2023-11-01T04:16:32.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628555+0000", 2023-11-01T04:16:32.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000142783 2023-11-01T04:16:32.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628587+0000", 2023-11-01T04:16:32.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2026999999999997e-05 2023-11-01T04:16:32.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831677+0000", 2023-11-01T04:16:32.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20309039200000001 2023-11-01T04:16:32.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831761+0000", 2023-11-01T04:16:32.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3461999999999995e-05 2023-11-01T04:16:32.574 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.575 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.575 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.575 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:55 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'33, mlcod=26'29)", 2023-11-01T04:16:32.575 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:01.629257+0000", 2023-11-01T04:16:32.576 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.897848333999999, 2023-11-01T04:16:32.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20259303100000001, 2023-11-01T04:16:32.576 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.576 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.576 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629257+0000", 2023-11-01T04:16:32.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629257+0000", 2023-11-01T04:16:32.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629259+0000", 2023-11-01T04:16:32.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8530000000000001e-06 2023-11-01T04:16:32.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629358+0000", 2023-11-01T04:16:32.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9019999999999997e-05 2023-11-01T04:16:32.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629386+0000", 2023-11-01T04:16:32.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8453999999999999e-05 2023-11-01T04:16:32.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629401+0000", 2023-11-01T04:16:32.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4465e-05 2023-11-01T04:16:32.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.829838+0000", 2023-11-01T04:16:32.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20043709100000001 2023-11-01T04:16:32.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.829873+0000", 2023-11-01T04:16:32.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5253000000000003e-05 2023-11-01T04:16:32.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.584 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831769+0000", 2023-11-01T04:16:32.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018957100000000001 2023-11-01T04:16:32.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831850+0000", 2023-11-01T04:16:32.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1185000000000003e-05 2023-11-01T04:16:32.585 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.586 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.586 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.586 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:104 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'66, mlcod=28'64)", 2023-11-01T04:16:32.587 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.140797+0000", 2023-11-01T04:16:32.587 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.386308042, 2023-11-01T04:16:32.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20170181600000001, 2023-11-01T04:16:32.587 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.587 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.587 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.140797+0000", 2023-11-01T04:16:32.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.140797+0000", 2023-11-01T04:16:32.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.590 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.140799+0000", 2023-11-01T04:16:32.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.176e-06 2023-11-01T04:16:32.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.140813+0000", 2023-11-01T04:16:32.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3791999999999999e-05 2023-11-01T04:16:32.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.140846+0000", 2023-11-01T04:16:32.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2824000000000003e-05 2023-11-01T04:16:32.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.240986+0000", 2023-11-01T04:16:32.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100140625 2023-11-01T04:16:32.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241142+0000", 2023-11-01T04:16:32.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015555899999999999 2023-11-01T04:16:32.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241199+0000", 2023-11-01T04:16:32.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7321999999999998e-05 2023-11-01T04:16:32.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.342429+0000", 2023-11-01T04:16:32.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101229401 2023-11-01T04:16:32.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.342499+0000", 2023-11-01T04:16:32.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0117000000000006e-05 2023-11-01T04:16:32.596 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.597 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.597 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.597 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:105 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'67, mlcod=28'64)", 2023-11-01T04:16:32.597 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.241023+0000", 2023-11-01T04:16:32.598 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.286081606, 2023-11-01T04:16:32.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10730198000000001, 2023-11-01T04:16:32.598 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.598 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.598 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241023+0000", 2023-11-01T04:16:32.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.600 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241023+0000", 2023-11-01T04:16:32.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241024+0000", 2023-11-01T04:16:32.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9999999999999995e-07 2023-11-01T04:16:32.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241045+0000", 2023-11-01T04:16:32.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0684999999999999e-05 2023-11-01T04:16:32.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.602 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.603 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241062+0000", 2023-11-01T04:16:32.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6956000000000002e-05 2023-11-01T04:16:32.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.603 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.603 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241072+0000", 2023-11-01T04:16:32.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0329000000000001e-05 2023-11-01T04:16:32.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.604 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.341337+0000", 2023-11-01T04:16:32.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100264753 2023-11-01T04:16:32.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.341371+0000", 2023-11-01T04:16:32.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4241e-05 2023-11-01T04:16:32.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.348245+0000", 2023-11-01T04:16:32.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068737700000000004 2023-11-01T04:16:32.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.348325+0000", 2023-11-01T04:16:32.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0246000000000004e-05 2023-11-01T04:16:32.607 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.607 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.608 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.608 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:106 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'68, mlcod=28'64)", 2023-11-01T04:16:32.608 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.241117+0000", 2023-11-01T04:16:32.609 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.285987624000001, 2023-11-01T04:16:32.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10725122099999999, 2023-11-01T04:16:32.609 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.609 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.609 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241117+0000", 2023-11-01T04:16:32.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241117+0000", 2023-11-01T04:16:32.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241118+0000", 2023-11-01T04:16:32.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8400000000000004e-07 2023-11-01T04:16:32.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241174+0000", 2023-11-01T04:16:32.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6097999999999997e-05 2023-11-01T04:16:32.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241190+0000", 2023-11-01T04:16:32.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5545000000000001e-05 2023-11-01T04:16:32.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241201+0000", 2023-11-01T04:16:32.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1148999999999999e-05 2023-11-01T04:16:32.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.341896+0000", 2023-11-01T04:16:32.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100695009 2023-11-01T04:16:32.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.341927+0000", 2023-11-01T04:16:32.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1597000000000003e-05 2023-11-01T04:16:32.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.348328+0000", 2023-11-01T04:16:32.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006401042 2023-11-01T04:16:32.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.348369+0000", 2023-11-01T04:16:32.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0197000000000002e-05 2023-11-01T04:16:32.618 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.618 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.619 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.619 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:107 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'69, mlcod=28'64)", 2023-11-01T04:16:32.619 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.241214+0000", 2023-11-01T04:16:32.620 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.285890639000002, 2023-11-01T04:16:32.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.109683828, 2023-11-01T04:16:32.620 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.620 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.620 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241214+0000", 2023-11-01T04:16:32.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241214+0000", 2023-11-01T04:16:32.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241215+0000", 2023-11-01T04:16:32.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7999999999999995e-07 2023-11-01T04:16:32.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241224+0000", 2023-11-01T04:16:32.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1630000000000002e-06 2023-11-01T04:16:32.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241233+0000", 2023-11-01T04:16:32.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3520000000000007e-06 2023-11-01T04:16:32.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241239+0000", 2023-11-01T04:16:32.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1890000000000004e-06 2023-11-01T04:16:32.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.342619+0000", 2023-11-01T04:16:32.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101380392 2023-11-01T04:16:32.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.627 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.627 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.342679+0000", 2023-11-01T04:16:32.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0387999999999998e-05 2023-11-01T04:16:32.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.628 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350812+0000", 2023-11-01T04:16:32.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0081325249999999998 2023-11-01T04:16:32.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350898+0000", 2023-11-01T04:16:32.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6238999999999999e-05 2023-11-01T04:16:32.629 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.630 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.630 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.630 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:108 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'70, mlcod=28'64)", 2023-11-01T04:16:32.631 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.242130+0000", 2023-11-01T04:16:32.631 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.284974940000001, 2023-11-01T04:16:32.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108814548, 2023-11-01T04:16:32.631 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.631 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.632 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242130+0000", 2023-11-01T04:16:32.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242130+0000", 2023-11-01T04:16:32.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242132+0000", 2023-11-01T04:16:32.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7799999999999999e-06 2023-11-01T04:16:32.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242144+0000", 2023-11-01T04:16:32.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2201999999999999e-05 2023-11-01T04:16:32.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242174+0000", 2023-11-01T04:16:32.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9484000000000001e-05 2023-11-01T04:16:32.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242192+0000", 2023-11-01T04:16:32.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8862e-05 2023-11-01T04:16:32.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.343513+0000", 2023-11-01T04:16:32.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101320113 2023-11-01T04:16:32.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.343569+0000", 2023-11-01T04:16:32.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6158e-05 2023-11-01T04:16:32.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350902+0000", 2023-11-01T04:16:32.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0073331500000000001 2023-11-01T04:16:32.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350945+0000", 2023-11-01T04:16:32.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2799000000000001e-05 2023-11-01T04:16:32.641 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.641 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.641 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.642 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:109 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'71, mlcod=28'64)", 2023-11-01T04:16:32.642 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.243545+0000", 2023-11-01T04:16:32.642 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.283560348999998, 2023-11-01T04:16:32.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.107439464, 2023-11-01T04:16:32.642 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.643 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.643 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243545+0000", 2023-11-01T04:16:32.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243545+0000", 2023-11-01T04:16:32.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243547+0000", 2023-11-01T04:16:32.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8530000000000001e-06 2023-11-01T04:16:32.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243603+0000", 2023-11-01T04:16:32.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6498e-05 2023-11-01T04:16:32.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243632+0000", 2023-11-01T04:16:32.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9135000000000001e-05 2023-11-01T04:16:32.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243646+0000", 2023-11-01T04:16:32.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3845e-05 2023-11-01T04:16:32.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.344360+0000", 2023-11-01T04:16:32.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10071392699999999 2023-11-01T04:16:32.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.344410+0000", 2023-11-01T04:16:32.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0078999999999999e-05 2023-11-01T04:16:32.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350947+0000", 2023-11-01T04:16:32.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0065365129999999999 2023-11-01T04:16:32.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350984+0000", 2023-11-01T04:16:32.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7614000000000003e-05 2023-11-01T04:16:32.652 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.652 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.652 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.653 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:110 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'72, mlcod=28'64)", 2023-11-01T04:16:32.653 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.244933+0000", 2023-11-01T04:16:32.653 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.282172381999999, 2023-11-01T04:16:32.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.107682845, 2023-11-01T04:16:32.653 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.653 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.654 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244933+0000", 2023-11-01T04:16:32.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244933+0000", 2023-11-01T04:16:32.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244934+0000", 2023-11-01T04:16:32.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.776e-06 2023-11-01T04:16:32.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244947+0000", 2023-11-01T04:16:32.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2046e-05 2023-11-01T04:16:32.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244976+0000", 2023-11-01T04:16:32.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9141000000000001e-05 2023-11-01T04:16:32.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244989+0000", 2023-11-01T04:16:32.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3675e-05 2023-11-01T04:16:32.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.345397+0000", 2023-11-01T04:16:32.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100407617 2023-11-01T04:16:32.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.345451+0000", 2023-11-01T04:16:32.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4475999999999998e-05 2023-11-01T04:16:32.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.352527+0000", 2023-11-01T04:16:32.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070755089999999998 2023-11-01T04:16:32.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.352616+0000", 2023-11-01T04:16:32.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8604999999999996e-05 2023-11-01T04:16:32.662 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.663 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.663 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.663 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:111 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'73, mlcod=28'64)", 2023-11-01T04:16:32.663 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.246238+0000", 2023-11-01T04:16:32.664 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.280866831000001, 2023-11-01T04:16:32.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10642051399999999, 2023-11-01T04:16:32.664 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.664 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.664 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.246238+0000", 2023-11-01T04:16:32.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.246238+0000", 2023-11-01T04:16:32.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.246240+0000", 2023-11-01T04:16:32.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.776e-06 2023-11-01T04:16:32.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.246252+0000", 2023-11-01T04:16:32.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1671999999999999e-05 2023-11-01T04:16:32.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.246280+0000", 2023-11-01T04:16:32.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8551000000000002e-05 2023-11-01T04:16:32.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.246294+0000", 2023-11-01T04:16:32.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3583e-05 2023-11-01T04:16:32.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.346176+0000", 2023-11-01T04:16:32.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.09988242 2023-11-01T04:16:32.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.346213+0000", 2023-11-01T04:16:32.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7209999999999998e-05 2023-11-01T04:16:32.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.352619+0000", 2023-11-01T04:16:32.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064055220000000003 2023-11-01T04:16:32.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.352659+0000", 2023-11-01T04:16:32.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9780000000000002e-05 2023-11-01T04:16:32.673 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.673 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.674 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.674 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:118 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'80, mlcod=28'78)", 2023-11-01T04:16:32.674 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.860484+0000", 2023-11-01T04:16:32.675 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 26.666620728000002, 2023-11-01T04:16:32.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20313931199999999, 2023-11-01T04:16:32.675 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.675 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.675 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860484+0000", 2023-11-01T04:16:32.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860484+0000", 2023-11-01T04:16:32.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860487+0000", 2023-11-01T04:16:32.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5239999999999999e-06 2023-11-01T04:16:32.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860560+0000", 2023-11-01T04:16:32.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3109999999999996e-05 2023-11-01T04:16:32.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860593+0000", 2023-11-01T04:16:32.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3077000000000002e-05 2023-11-01T04:16:32.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860611+0000", 2023-11-01T04:16:32.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7881999999999999e-05 2023-11-01T04:16:32.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.960884+0000", 2023-11-01T04:16:32.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100273563 2023-11-01T04:16:32.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.960949+0000", 2023-11-01T04:16:32.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4250000000000003e-05 2023-11-01T04:16:32.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063550+0000", 2023-11-01T04:16:32.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.102601553 2023-11-01T04:16:32.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063624+0000", 2023-11-01T04:16:32.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3353000000000001e-05 2023-11-01T04:16:32.684 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.684 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.684 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.685 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:119 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'81, mlcod=28'78)", 2023-11-01T04:16:32.685 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.861924+0000", 2023-11-01T04:16:32.685 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 26.665181552, 2023-11-01T04:16:32.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.201754672, 2023-11-01T04:16:32.686 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.686 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.686 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.861924+0000", 2023-11-01T04:16:32.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.861924+0000", 2023-11-01T04:16:32.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.861925+0000", 2023-11-01T04:16:32.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.663e-06 2023-11-01T04:16:32.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.862002+0000", 2023-11-01T04:16:32.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7217000000000003e-05 2023-11-01T04:16:32.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.862024+0000", 2023-11-01T04:16:32.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1957999999999999e-05 2023-11-01T04:16:32.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.691 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.862035+0000", 2023-11-01T04:16:32.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0482000000000001e-05 2023-11-01T04:16:32.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.961997+0000", 2023-11-01T04:16:32.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.099962336999999998 2023-11-01T04:16:32.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.962032+0000", 2023-11-01T04:16:32.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4804000000000001e-05 2023-11-01T04:16:32.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063634+0000", 2023-11-01T04:16:32.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101602151 2023-11-01T04:16:32.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063678+0000", 2023-11-01T04:16:32.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4060000000000002e-05 2023-11-01T04:16:32.695 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.695 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.695 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.696 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:120 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'82, mlcod=28'78)", 2023-11-01T04:16:32.696 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.863328+0000", 2023-11-01T04:16:32.696 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 26.663777079999999, 2023-11-01T04:16:32.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20039775900000001, 2023-11-01T04:16:32.696 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.697 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.697 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863328+0000", 2023-11-01T04:16:32.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863328+0000", 2023-11-01T04:16:32.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863330+0000", 2023-11-01T04:16:32.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7770000000000001e-06 2023-11-01T04:16:32.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863442+0000", 2023-11-01T04:16:32.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112596 2023-11-01T04:16:32.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863472+0000", 2023-11-01T04:16:32.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9490999999999999e-05 2023-11-01T04:16:32.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863486+0000", 2023-11-01T04:16:32.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3689000000000001e-05 2023-11-01T04:16:32.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.962703+0000", 2023-11-01T04:16:32.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.099217692999999996 2023-11-01T04:16:32.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.962756+0000", 2023-11-01T04:16:32.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2454999999999998e-05 2023-11-01T04:16:32.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063680+0000", 2023-11-01T04:16:32.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100924298 2023-11-01T04:16:32.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063726+0000", 2023-11-01T04:16:32.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5760000000000002e-05 2023-11-01T04:16:32.706 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.706 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.706 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.707 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:121 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'83, mlcod=28'78)", 2023-11-01T04:16:32.707 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.864836+0000", 2023-11-01T04:16:32.707 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 26.662269254000002, 2023-11-01T04:16:32.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20010170799999999, 2023-11-01T04:16:32.707 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.708 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.708 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864836+0000", 2023-11-01T04:16:32.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864836+0000", 2023-11-01T04:16:32.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864838+0000", 2023-11-01T04:16:32.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7999999999999999e-06 2023-11-01T04:16:32.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864956+0000", 2023-11-01T04:16:32.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000118293 2023-11-01T04:16:32.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864985+0000", 2023-11-01T04:16:32.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9187999999999999e-05 2023-11-01T04:16:32.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864999+0000", 2023-11-01T04:16:32.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3718e-05 2023-11-01T04:16:32.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.062876+0000", 2023-11-01T04:16:32.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.197877154 2023-11-01T04:16:32.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.062909+0000", 2023-11-01T04:16:32.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3059999999999999e-05 2023-11-01T04:16:32.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.064827+0000", 2023-11-01T04:16:32.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019179209999999999 2023-11-01T04:16:32.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.716 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.064938+0000", 2023-11-01T04:16:32.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110574 2023-11-01T04:16:32.717 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.717 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.717 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.718 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:133 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'91, mlcod=28'85)", 2023-11-01T04:16:32.718 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:06.724959+0000", 2023-11-01T04:16:32.718 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.802145755000002, 2023-11-01T04:16:32.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108020745, 2023-11-01T04:16:32.718 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.718 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.719 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.724959+0000", 2023-11-01T04:16:32.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.724959+0000", 2023-11-01T04:16:32.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.724960+0000", 2023-11-01T04:16:32.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3000000000000001e-07 2023-11-01T04:16:32.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.724965+0000", 2023-11-01T04:16:32.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.463e-06 2023-11-01T04:16:32.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.724978+0000", 2023-11-01T04:16:32.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2748999999999999e-05 2023-11-01T04:16:32.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.825118+0000", 2023-11-01T04:16:32.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100140121 2023-11-01T04:16:32.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.827391+0000", 2023-11-01T04:16:32.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022731800000000001 2023-11-01T04:16:32.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.827428+0000", 2023-11-01T04:16:32.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.65e-05 2023-11-01T04:16:32.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.832911+0000", 2023-11-01T04:16:32.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0054829340000000001 2023-11-01T04:16:32.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.832980+0000", 2023-11-01T04:16:32.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9368000000000004e-05 2023-11-01T04:16:32.728 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.728 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.728 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.729 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:138 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'96, mlcod=28'85)", 2023-11-01T04:16:32.729 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:06.925635+0000", 2023-11-01T04:16:32.729 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.601469683000001, 2023-11-01T04:16:32.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20470132099999999, 2023-11-01T04:16:32.730 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.730 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.730 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.925635+0000", 2023-11-01T04:16:32.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.925635+0000", 2023-11-01T04:16:32.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.925636+0000", 2023-11-01T04:16:32.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1999999999999999e-07 2023-11-01T04:16:32.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.925647+0000", 2023-11-01T04:16:32.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1348999999999999e-05 2023-11-01T04:16:32.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.925660+0000", 2023-11-01T04:16:32.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2782000000000001e-05 2023-11-01T04:16:32.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.025800+0000", 2023-11-01T04:16:32.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100139379 2023-11-01T04:16:32.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.736 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.025960+0000", 2023-11-01T04:16:32.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000160063 2023-11-01T04:16:32.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.736 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.026036+0000", 2023-11-01T04:16:32.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6637000000000005e-05 2023-11-01T04:16:32.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.130231+0000", 2023-11-01T04:16:32.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.104194433 2023-11-01T04:16:32.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.130337+0000", 2023-11-01T04:16:32.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010605800000000001 2023-11-01T04:16:32.739 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.739 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.739 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.740 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:141 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'99, mlcod=28'86)", 2023-11-01T04:16:32.740 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:07.026042+0000", 2023-11-01T04:16:32.740 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.501062706999999, 2023-11-01T04:16:32.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10566460499999999, 2023-11-01T04:16:32.740 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:32.741 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:32.741 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:32.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:32.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.026042+0000", 2023-11-01T04:16:32.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:32.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.026042+0000", 2023-11-01T04:16:32.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:32.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:32.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.026043+0000", 2023-11-01T04:16:32.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1399999999999997e-07 2023-11-01T04:16:32.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.744 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:32.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.026050+0000", 2023-11-01T04:16:32.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2790000000000001e-06 2023-11-01T04:16:32.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:32.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.026059+0000", 2023-11-01T04:16:32.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6219999999999998e-06 2023-11-01T04:16:32.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:32.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.026066+0000", 2023-11-01T04:16:32.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2799999999999998e-06 2023-11-01T04:16:32.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.747 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:32.747 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.129441+0000", 2023-11-01T04:16:32.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.103374547 2023-11-01T04:16:32.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:32.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.129502+0000", 2023-11-01T04:16:32.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1644999999999997e-05 2023-11-01T04:16:32.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.750 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:32.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.131642+0000", 2023-11-01T04:16:32.751 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021401530000000001 2023-11-01T04:16:32.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:32.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:32.751 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:32.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.131707+0000", 2023-11-01T04:16:32.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4565000000000003e-05 2023-11-01T04:16:32.752 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.753 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.753 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.753 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:32.754 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:32.754 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:32.756 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:16:32.754+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:16:32.756 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-01T04:16:32.857 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:32.854+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:32.886 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:32.886 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:16:32.886 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2023-11-01T04:16:32.887 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:32.900 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-01T04:16:32.957 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:32.954+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:33.020 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:33.020 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:16:33.020 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2023-11-01T04:16:33.020 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2023-11-01T04:16:33.020 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:33.034 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-01T04:16:33.058 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:33.055+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:33.088 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.13 to 1 (10000) 2023-11-01T04:16:33.098 INFO:tasks.rados.rados.0.smithi190.stdout:158: done (0 left) 2023-11-01T04:16:33.098 INFO:tasks.rados.rados.0.smithi190.stdout:159: read oid 42 snap -1 2023-11-01T04:16:33.099 INFO:tasks.rados.rados.0.smithi190.stdout:159: expect (ObjNum 71 snap 8 seq_num 71) 2023-11-01T04:16:33.099 INFO:tasks.rados.rados.0.smithi190.stdout:160: rmattr oid 17 current snap is 13 2023-11-01T04:16:33.099 INFO:tasks.rados.rados.0.smithi190.stdout:160: done (1 left) 2023-11-01T04:16:33.099 INFO:tasks.rados.rados.0.smithi190.stdout:161: setattr oid 28 current snap is 13 2023-11-01T04:16:33.104 INFO:tasks.rados.rados.0.smithi190.stdout:162: copy_from oid 33 from oid 16 current snap is 13 2023-11-01T04:16:33.104 INFO:tasks.rados.rados.0.smithi190.stdout:163: setattr oid 32 current snap is 13 2023-11-01T04:16:33.111 INFO:tasks.rados.rados.0.smithi190.stdout:164: snap_remove snap 6 2023-11-01T04:16:33.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2023-11-01T04:16:33.159 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:33.155+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:33.167 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2023-11-01T04:16:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2023-11-01T04:16:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:16:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.168 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:188 3.fs0 3:f81e983c:::smithi19039307-34:head [create,setxattr XNeZS[lOPtVca\\Pa[\\rdfbinb[bcjhtYSpmcafZm (1928) in=40b,setxattr TfflYYqPnps]lo^qSlp___TitNdd`]kaPTVOTcsh (1648) in=40b,setxattr TNb_onpY[j_dkhfmnWtm]Yktg]]RNtbQe[OUTNlX (770) in=40b,setxattr ZcgnkhY\\orTYaZh]PPldj`SNq`p\\cX`k]h\\bZY_Y (700) in=40b,setxattr gPQVnqhpsVhW_kq`QnafXrlRZtZ\\YnRftgQTishR (595) in=40b,setxattr U`XqYi[WhoqnpSPRc\\RhnppZp^qY\\qUfjpqs[r^a (1892) in=40b,setxattr cTmhd`pWgrRf_VfVgOtgeOP^ShP]lsOtpoTj[\\gl (418) in=40b,setxattr aa_taYTfslg`lRRrOkkraN\\XiW^V\\ljl]NqXin`a (314) in=40b,setxattr ibl]`igtgVThY^Qse\\okYYUtTmolX\\W]p]Q[P\\il (1403) in=40b,setxattr \\ZabSO[UUqeZa`hddNh[ZjaRRgUZoQtkn\\QdsX]W (1541) in=40b,setxattr jUkQUPTNbo`rZNmYlUmhb]ojc]aeheoVRfp]bPWf (5052) in=40b,setxattr Ofj`oQ^cR]da^SPsTaXmmPse\\OZ`VSqQ]U`efsZl (419) in=40b,setxattr a[cdkWos\\OrUVPciUaa^_^`lWVTScffiWkd\\[kgf (1261) in=40b,setxattr jXb]\\WhbatdZ[qTTiZO]kPd^`l[kajOYiQ[iqSYc (244) in=40b,setxattr \\im]jQhbanUn\\_^h`jaU]W`ReaZN_je[]kgg (1578) in=36b,setxattr ZbosoRORS^_de]rN^_daOiha\\Q\\eRae^[sQUjXsV (9120) in=40b,setxattr gcn]XpZUkOZr\\pkrVVZpYiR\\alh_PaZa_S[PYcn] (384) in=40b,setxattr \\^qUOaNXSTjtmafhfbo\\gseTTkV`mgnacYq^bkcc (34) in=40b,setxattr iUa^mOQ]TqjVSUU_nXfY]f`NjljiiaYbrm[o]aes (1611) in=40b,setxattr tXaQPQbc^aiV[^reVOPpib^caoWf^XgqilO^_g[k (1016) in=40b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:33.169 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.964946+0000", 2023-11-01T04:16:33.169 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 16.202019921000002, 2023-11-01T04:16:33.169 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.309617849, 2023-11-01T04:16:33.169 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.169 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 188 2023-11-01T04:16:33.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.964946+0000", 2023-11-01T04:16:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.964946+0000", 2023-11-01T04:16:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.964948+0000", 2023-11-01T04:16:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.824e-06 2023-11-01T04:16:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.173 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.965026+0000", 2023-11-01T04:16:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.8151000000000004e-05 2023-11-01T04:16:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.174 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.965047+0000", 2023-11-01T04:16:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0891e-05 2023-11-01T04:16:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.175 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.965062+0000", 2023-11-01T04:16:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5310000000000001e-05 2023-11-01T04:16:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.176 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.965154+0000", 2023-11-01T04:16:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2163999999999994e-05 2023-11-01T04:16:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2023-11-01T04:16:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.965165+0000", 2023-11-01T04:16:33.177 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0703000000000001e-05 2023-11-01T04:16:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.167756+0000", 2023-11-01T04:16:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20259100399999999 2023-11-01T04:16:33.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.179 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.179 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:33.179 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.167952+0000", 2023-11-01T04:16:33.179 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00019548100000000001 2023-11-01T04:16:33.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.169709+0000", 2023-11-01T04:16:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0017572989999999999 2023-11-01T04:16:33.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.171668+0000", 2023-11-01T04:16:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0019592659999999999 2023-11-01T04:16:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.181 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:16:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.274435+0000", 2023-11-01T04:16:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10276656000000001 2023-11-01T04:16:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.182 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.274564+0000", 2023-11-01T04:16:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012919599999999999 2023-11-01T04:16:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.183 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 37/31 ECSubWriteReply(tid=9, last_complete=37'4, committed=1, applied=1))", 2023-11-01T04:16:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:17.277556+0000", 2023-11-01T04:16:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 15.889409556, 2023-11-01T04:16:33.184 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20033701600000001, 2023-11-01T04:16:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.185 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.277556+0000", 2023-11-01T04:16:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.277556+0000", 2023-11-01T04:16:33.186 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.277558+0000", 2023-11-01T04:16:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.217e-06 2023-11-01T04:16:33.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.277571+0000", 2023-11-01T04:16:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3032e-05 2023-11-01T04:16:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.188 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.277593+0000", 2023-11-01T04:16:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2160000000000001e-05 2023-11-01T04:16:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.377685+0000", 2023-11-01T04:16:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10009243700000001 2023-11-01T04:16:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.190 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.477859+0000", 2023-11-01T04:16:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10017401500000001 2023-11-01T04:16:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.191 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.477894+0000", 2023-11-01T04:16:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4155000000000001e-05 2023-11-01T04:16:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(osd.11.22:2 3.cs0 3:33c6488a:::smithi19039307-6:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:17.378858+0000", 2023-11-01T04:16:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 15.788108165000001, 2023-11-01T04:16:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.203072057, 2023-11-01T04:16:33.193 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378858+0000", 2023-11-01T04:16:33.194 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378858+0000", 2023-11-01T04:16:33.195 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378860+0000", 2023-11-01T04:16:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5339999999999998e-06 2023-11-01T04:16:33.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378872+0000", 2023-11-01T04:16:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1365999999999999e-05 2023-11-01T04:16:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.197 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378896+0000", 2023-11-01T04:16:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3935999999999999e-05 2023-11-01T04:16:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.198 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378909+0000", 2023-11-01T04:16:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3523000000000001e-05 2023-11-01T04:16:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.199 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.379027+0000", 2023-11-01T04:16:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000117963 2023-11-01T04:16:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.379186+0000", 2023-11-01T04:16:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000158939 2023-11-01T04:16:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.201 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.581930+0000", 2023-11-01T04:16:33.202 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.202743796 2023-11-01T04:16:33.202 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.202 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.202 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.203 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.203 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 41/31 ECSubWriteReply(tid=12, last_complete=41'6, committed=1, applied=1))", 2023-11-01T04:16:33.203 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.240162+0000", 2023-11-01T04:16:33.203 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 11.926804133999999, 2023-11-01T04:16:33.203 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19880108899999999, 2023-11-01T04:16:33.203 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.204 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.204 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.204 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.204 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.204 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.240162+0000", 2023-11-01T04:16:33.204 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.240162+0000", 2023-11-01T04:16:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.240163+0000", 2023-11-01T04:16:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4200000000000004e-07 2023-11-01T04:16:33.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.240168+0000", 2023-11-01T04:16:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4729999999999999e-06 2023-11-01T04:16:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.207 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.240181+0000", 2023-11-01T04:16:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2639999999999999e-05 2023-11-01T04:16:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.208 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.209 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.340265+0000", 2023-11-01T04:16:33.209 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100084464 2023-11-01T04:16:33.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.209 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.209 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.438933+0000", 2023-11-01T04:16:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.098667650999999995 2023-11-01T04:16:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.210 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.211 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.438963+0000", 2023-11-01T04:16:33.211 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9918999999999999e-05 2023-11-01T04:16:33.211 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.211 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.211 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(osd.14.22:2 3.cs0 3:33c6488a:::smithi19039307-6:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.134094+0000", 2023-11-01T04:16:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 7.0328718209999996, 2023-11-01T04:16:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.35681233299999998, 2023-11-01T04:16:33.212 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134094+0000", 2023-11-01T04:16:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.214 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.214 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.214 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134094+0000", 2023-11-01T04:16:33.214 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134096+0000", 2023-11-01T04:16:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1919999999999999e-06 2023-11-01T04:16:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.215 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134108+0000", 2023-11-01T04:16:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1508000000000001e-05 2023-11-01T04:16:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.216 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134129+0000", 2023-11-01T04:16:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0968e-05 2023-11-01T04:16:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134151+0000", 2023-11-01T04:16:33.217 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2480000000000002e-05 2023-11-01T04:16:33.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.218 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.218 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.218 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134288+0000", 2023-11-01T04:16:33.218 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013673799999999999 2023-11-01T04:16:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134380+0000", 2023-11-01T04:16:33.219 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.1555e-05 2023-11-01T04:16:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.490907+0000", 2023-11-01T04:16:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.35652689199999998 2023-11-01T04:16:33.220 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.cs0 45/31 ECSubReadReply(tid=18, attrs_read=0))", 2023-11-01T04:16:33.221 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.135072+0000", 2023-11-01T04:16:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 7.0318945069999996, 2023-11-01T04:16:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20028368799999999, 2023-11-01T04:16:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.222 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.223 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.223 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135072+0000", 2023-11-01T04:16:33.223 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.223 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.223 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135072+0000", 2023-11-01T04:16:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135072+0000", 2023-11-01T04:16:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5199999999999995e-07 2023-11-01T04:16:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135109+0000", 2023-11-01T04:16:33.225 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6155999999999997e-05 2023-11-01T04:16:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135118+0000", 2023-11-01T04:16:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7399999999999999e-06 2023-11-01T04:16:33.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135123+0000", 2023-11-01T04:16:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.8160000000000001e-06 2023-11-01T04:16:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.227 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.335285+0000", 2023-11-01T04:16:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20016213599999999 2023-11-01T04:16:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.228 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.335355+0000", 2023-11-01T04:16:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9888000000000006e-05 2023-11-01T04:16:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.229 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.cs0 46/31 ECSubReadReply(tid=18, attrs_read=0))", 2023-11-01T04:16:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.135473+0000", 2023-11-01T04:16:33.230 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 7.031492675, 2023-11-01T04:16:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19999888900000001, 2023-11-01T04:16:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.231 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135473+0000", 2023-11-01T04:16:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.232 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135473+0000", 2023-11-01T04:16:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135474+0000", 2023-11-01T04:16:33.233 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1200000000000002e-07 2023-11-01T04:16:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135586+0000", 2023-11-01T04:16:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000112274 2023-11-01T04:16:33.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135602+0000", 2023-11-01T04:16:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5044e-05 2023-11-01T04:16:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.235 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135611+0000", 2023-11-01T04:16:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0210000000000005e-06 2023-11-01T04:16:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.236 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.335430+0000", 2023-11-01T04:16:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19981938099999999 2023-11-01T04:16:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.237 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.335472+0000", 2023-11-01T04:16:33.238 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2357000000000001e-05 2023-11-01T04:16:33.238 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.238 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.238 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.239 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.239 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s2 46/31 ECSubWrite(tid=235, reqid=client.4559.0:329, at_version=46'35, trim_to=0'0, roll_forward_to=45'33))", 2023-11-01T04:16:33.239 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.179884+0000", 2023-11-01T04:16:33.239 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.9870816080000004, 2023-11-01T04:16:33.239 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20567904100000001, 2023-11-01T04:16:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.240 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.179884+0000", 2023-11-01T04:16:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.179884+0000", 2023-11-01T04:16:33.241 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.179885+0000", 2023-11-01T04:16:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.74e-07 2023-11-01T04:16:33.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.180252+0000", 2023-11-01T04:16:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00036695899999999998 2023-11-01T04:16:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.243 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.180264+0000", 2023-11-01T04:16:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2850999999999999e-05 2023-11-01T04:16:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.244 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.280394+0000", 2023-11-01T04:16:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100129722 2023-11-01T04:16:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.245 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.280540+0000", 2023-11-01T04:16:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014556000000000001 2023-11-01T04:16:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.280557+0000", 2023-11-01T04:16:33.246 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7543000000000001e-05 2023-11-01T04:16:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.385509+0000", 2023-11-01T04:16:33.247 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104951378 2023-11-01T04:16:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.385563+0000", 2023-11-01T04:16:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4854e-05 2023-11-01T04:16:33.248 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.249 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.249 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.249 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.249 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.249 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.8s5 46/31 ECSubWrite(tid=30, reqid=client.4559.0:341, at_version=46'22, trim_to=0'0, roll_forward_to=43'14))", 2023-11-01T04:16:33.250 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.266862+0000", 2023-11-01T04:16:33.250 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.9001035860000002, 2023-11-01T04:16:33.250 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20203428600000001, 2023-11-01T04:16:33.250 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.250 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266862+0000", 2023-11-01T04:16:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266862+0000", 2023-11-01T04:16:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.252 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266865+0000", 2023-11-01T04:16:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2840000000000001e-06 2023-11-01T04:16:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.253 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266897+0000", 2023-11-01T04:16:33.254 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2743000000000001e-05 2023-11-01T04:16:33.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.254 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.254 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266920+0000", 2023-11-01T04:16:33.255 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2878999999999999e-05 2023-11-01T04:16:33.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.255 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.255 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.255 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266939+0000", 2023-11-01T04:16:33.255 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8185999999999998e-05 2023-11-01T04:16:33.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.256 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.256 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.256 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.467334+0000", 2023-11-01T04:16:33.256 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20039552699999999 2023-11-01T04:16:33.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.256 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.257 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.257 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.467350+0000", 2023-11-01T04:16:33.257 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5682999999999999e-05 2023-11-01T04:16:33.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.257 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.258 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.258 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.468855+0000", 2023-11-01T04:16:33.258 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001504825 2023-11-01T04:16:33.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.258 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.468897+0000", 2023-11-01T04:16:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2159000000000001e-05 2023-11-01T04:16:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.259 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.260 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.260 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s2 46/31 ECSubWrite(tid=236, reqid=client.4559.0:330, at_version=46'36, trim_to=0'0, roll_forward_to=45'33))", 2023-11-01T04:16:33.260 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.282526+0000", 2023-11-01T04:16:33.260 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.8844401900000003, 2023-11-01T04:16:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10378303999999999, 2023-11-01T04:16:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.261 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.262 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282526+0000", 2023-11-01T04:16:33.262 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.262 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.262 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.262 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282526+0000", 2023-11-01T04:16:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282528+0000", 2023-11-01T04:16:33.263 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7099999999999999e-06 2023-11-01T04:16:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282629+0000", 2023-11-01T04:16:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010132399999999999 2023-11-01T04:16:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.264 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.265 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.265 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282658+0000", 2023-11-01T04:16:33.265 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9496e-05 2023-11-01T04:16:33.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.265 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.265 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282676+0000", 2023-11-01T04:16:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7842999999999998e-05 2023-11-01T04:16:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.266 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282766+0000", 2023-11-01T04:16:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0056999999999995e-05 2023-11-01T04:16:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.267 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282777+0000", 2023-11-01T04:16:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0547000000000001e-05 2023-11-01T04:16:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.386263+0000", 2023-11-01T04:16:33.268 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103486419 2023-11-01T04:16:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.386309+0000", 2023-11-01T04:16:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5643999999999999e-05 2023-11-01T04:16:33.269 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.270 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.270 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.270 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.270 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s2 46/31 ECSubWrite(tid=237, reqid=client.4559.0:331, at_version=46'37, trim_to=0'0, roll_forward_to=45'33))", 2023-11-01T04:16:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.282715+0000", 2023-11-01T04:16:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.8842515280000001, 2023-11-01T04:16:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103639125, 2023-11-01T04:16:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.271 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.272 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.272 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282715+0000", 2023-11-01T04:16:33.272 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.273 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.273 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282715+0000", 2023-11-01T04:16:33.273 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.273 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.273 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.273 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.274 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282715+0000", 2023-11-01T04:16:33.274 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1299999999999999e-07 2023-11-01T04:16:33.274 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.274 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.274 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.274 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282744+0000", 2023-11-01T04:16:33.275 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8609e-05 2023-11-01T04:16:33.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.275 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.275 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.275 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282765+0000", 2023-11-01T04:16:33.275 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1557e-05 2023-11-01T04:16:33.276 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.276 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.276 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.276 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282782+0000", 2023-11-01T04:16:33.276 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6599000000000002e-05 2023-11-01T04:16:33.276 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.277 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.277 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.277 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.283338+0000", 2023-11-01T04:16:33.277 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00055595999999999998 2023-11-01T04:16:33.277 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.277 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.278 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.278 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.283353+0000", 2023-11-01T04:16:33.278 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4697e-05 2023-11-01T04:16:33.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.278 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.278 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.279 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.386315+0000", 2023-11-01T04:16:33.279 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10296182800000001 2023-11-01T04:16:33.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.279 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.280 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.386354+0000", 2023-11-01T04:16:33.280 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9062e-05 2023-11-01T04:16:33.280 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.280 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.280 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.281 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.281 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:347 3.fs0 3:f81e983c:::smithi19039307-34:c [read 0~557056,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:33.281 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.032054+0000", 2023-11-01T04:16:33.281 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.1349118850000002, 2023-11-01T04:16:33.281 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20755394999999999, 2023-11-01T04:16:33.281 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.282 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:33.282 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:33.282 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:33.282 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:33.282 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 347 2023-11-01T04:16:33.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.283 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.283 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.283 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.283 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032054+0000", 2023-11-01T04:16:33.283 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.283 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.284 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.284 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.284 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032054+0000", 2023-11-01T04:16:33.284 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.284 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.284 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.285 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.285 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032058+0000", 2023-11-01T04:16:33.285 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4740000000000001e-06 2023-11-01T04:16:33.285 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.285 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.286 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.286 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032074+0000", 2023-11-01T04:16:33.286 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.679e-05 2023-11-01T04:16:33.286 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.286 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.286 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.287 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032109+0000", 2023-11-01T04:16:33.287 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4309e-05 2023-11-01T04:16:33.287 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.287 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.287 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.287 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032128+0000", 2023-11-01T04:16:33.288 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8856999999999999e-05 2023-11-01T04:16:33.288 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.288 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.288 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.288 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032256+0000", 2023-11-01T04:16:33.288 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000128866 2023-11-01T04:16:33.289 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.289 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.289 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2023-11-01T04:16:33.289 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032268+0000", 2023-11-01T04:16:33.289 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1649e-05 2023-11-01T04:16:33.290 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.290 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.290 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.290 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.132806+0000", 2023-11-01T04:16:33.290 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10053780399999999 2023-11-01T04:16:33.291 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.291 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.291 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:33.291 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.132905+0000", 2023-11-01T04:16:33.291 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.9326000000000004e-05 2023-11-01T04:16:33.291 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.292 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.292 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.292 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.239608+0000", 2023-11-01T04:16:33.292 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.106702875 2023-11-01T04:16:33.292 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.292 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.293 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.293 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.293 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:348 3.fs0 3:f81e983c:::smithi19039307-34:c [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:33.293 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.032168+0000", 2023-11-01T04:16:33.293 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.1347975899999998, 2023-11-01T04:16:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30608133399999998, 2023-11-01T04:16:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:33.294 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 348 2023-11-01T04:16:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.295 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032168+0000", 2023-11-01T04:16:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.296 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032168+0000", 2023-11-01T04:16:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032169+0000", 2023-11-01T04:16:33.297 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8199999999999999e-07 2023-11-01T04:16:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032174+0000", 2023-11-01T04:16:33.298 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1379999999999998e-06 2023-11-01T04:16:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032186+0000", 2023-11-01T04:16:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1229e-05 2023-11-01T04:16:33.299 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032193+0000", 2023-11-01T04:16:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2749999999999998e-06 2023-11-01T04:16:33.300 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032361+0000", 2023-11-01T04:16:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016853199999999999 2023-11-01T04:16:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.301 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032368+0000", 2023-11-01T04:16:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.686e-06 2023-11-01T04:16:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.302 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.135190+0000", 2023-11-01T04:16:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10282246 2023-11-01T04:16:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:33.303 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.135261+0000", 2023-11-01T04:16:33.304 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0296999999999995e-05 2023-11-01T04:16:33.304 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.304 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.304 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.304 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.338250+0000", 2023-11-01T04:16:33.304 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20298903500000001 2023-11-01T04:16:33.305 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.305 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.305 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.305 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.305 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.305 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:349 3.fs0 3:f81e983c:::smithi19039307-34:c [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:33.305 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.032204+0000", 2023-11-01T04:16:33.306 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.1347621310000005, 2023-11-01T04:16:33.306 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.61100175800000001, 2023-11-01T04:16:33.306 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.306 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:33.306 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:33.307 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:33.307 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:33.307 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 349 2023-11-01T04:16:33.307 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.307 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.307 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.307 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.308 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032204+0000", 2023-11-01T04:16:33.308 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.308 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.308 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.308 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.308 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032204+0000", 2023-11-01T04:16:33.309 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.309 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.309 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.309 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.309 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032204+0000", 2023-11-01T04:16:33.310 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2200000000000004e-07 2023-11-01T04:16:33.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.310 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.310 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.310 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032209+0000", 2023-11-01T04:16:33.311 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5979999999999999e-06 2023-11-01T04:16:33.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.311 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.311 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.311 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032217+0000", 2023-11-01T04:16:33.311 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9019999999999999e-06 2023-11-01T04:16:33.312 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.312 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.312 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.312 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032223+0000", 2023-11-01T04:16:33.312 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7180000000000001e-06 2023-11-01T04:16:33.312 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.313 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.313 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.313 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.132534+0000", 2023-11-01T04:16:33.313 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10031093200000001 2023-11-01T04:16:33.313 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.313 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.314 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:33.314 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.132550+0000", 2023-11-01T04:16:33.314 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6269999999999998e-05 2023-11-01T04:16:33.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.314 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.314 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.315 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.339795+0000", 2023-11-01T04:16:33.315 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20724526200000001 2023-11-01T04:16:33.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.315 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.315 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:33.315 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.340097+0000", 2023-11-01T04:16:33.316 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00030174899999999997 2023-11-01T04:16:33.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.316 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.316 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.316 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.643206+0000", 2023-11-01T04:16:33.316 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30310880499999998 2023-11-01T04:16:33.317 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.317 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.317 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.317 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.317 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.fs0 47/31 ECSubReadReply(tid=21, attrs_read=0))", 2023-11-01T04:16:33.318 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.349538+0000", 2023-11-01T04:16:33.318 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.8174281360000002, 2023-11-01T04:16:33.318 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.29371594899999998, 2023-11-01T04:16:33.318 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.318 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.318 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.319 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.319 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.319 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349538+0000", 2023-11-01T04:16:33.319 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.320 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.320 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.320 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349538+0000", 2023-11-01T04:16:33.320 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.320 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.321 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.321 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349539+0000", 2023-11-01T04:16:33.321 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5370000000000001e-06 2023-11-01T04:16:33.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.321 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.321 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.322 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349659+0000", 2023-11-01T04:16:33.322 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011938 2023-11-01T04:16:33.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.322 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.322 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.323 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349686+0000", 2023-11-01T04:16:33.323 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.728e-05 2023-11-01T04:16:33.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.323 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.323 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.323 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349702+0000", 2023-11-01T04:16:33.324 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5693e-05 2023-11-01T04:16:33.324 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.324 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.324 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.324 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.642479+0000", 2023-11-01T04:16:33.324 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.29277763699999998 2023-11-01T04:16:33.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.325 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.325 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.643254+0000", 2023-11-01T04:16:33.325 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00077442199999999998 2023-11-01T04:16:33.325 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.326 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.326 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.326 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.326 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.3s3 47/31 ECSubWrite(tid=243, reqid=client.4559.0:358, at_version=47'13, trim_to=0'0, roll_forward_to=41'11))", 2023-11-01T04:16:33.326 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.508512+0000", 2023-11-01T04:16:33.327 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.6584535860000003, 2023-11-01T04:16:33.327 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104546521, 2023-11-01T04:16:33.327 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.327 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.327 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.327 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.328 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.328 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.508512+0000", 2023-11-01T04:16:33.328 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.328 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.328 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.329 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.508512+0000", 2023-11-01T04:16:33.329 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.329 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.329 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.330 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.508514+0000", 2023-11-01T04:16:33.330 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8700000000000001e-06 2023-11-01T04:16:33.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.330 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.330 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.330 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.508812+0000", 2023-11-01T04:16:33.331 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00029758600000000002 2023-11-01T04:16:33.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.331 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.331 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.508843+0000", 2023-11-01T04:16:33.331 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1046000000000001e-05 2023-11-01T04:16:33.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.332 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.332 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.332 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.608981+0000", 2023-11-01T04:16:33.332 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100138404 2023-11-01T04:16:33.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.333 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.333 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.333 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.609126+0000", 2023-11-01T04:16:33.333 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014511999999999999 2023-11-01T04:16:33.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.333 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.334 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.334 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.609142+0000", 2023-11-01T04:16:33.334 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5913000000000002e-05 2023-11-01T04:16:33.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.334 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.335 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.612960+0000", 2023-11-01T04:16:33.335 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.003817754 2023-11-01T04:16:33.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.335 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.335 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.335 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.613059+0000", 2023-11-01T04:16:33.336 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.8827999999999996e-05 2023-11-01T04:16:33.336 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.336 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.336 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.336 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.337 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds3 47/31 ECSubWrite(tid=76, reqid=client.4559.0:352, at_version=47'33, trim_to=0'0, roll_forward_to=46'31))", 2023-11-01T04:16:33.337 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.915243+0000", 2023-11-01T04:16:33.337 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.2517228620000003, 2023-11-01T04:16:33.337 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10447798699999999, 2023-11-01T04:16:33.337 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.337 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.338 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.338 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.338 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.915243+0000", 2023-11-01T04:16:33.338 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.339 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.339 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.915243+0000", 2023-11-01T04:16:33.339 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.340 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.340 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.915245+0000", 2023-11-01T04:16:33.340 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3e-06 2023-11-01T04:16:33.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.341 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.341 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.915681+0000", 2023-11-01T04:16:33.341 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00043578999999999997 2023-11-01T04:16:33.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.342 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.342 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.915710+0000", 2023-11-01T04:16:33.342 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8439e-05 2023-11-01T04:16:33.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.342 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.342 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.343 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.915729+0000", 2023-11-01T04:16:33.343 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9153000000000001e-05 2023-11-01T04:16:33.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.343 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.343 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.016005+0000", 2023-11-01T04:16:33.344 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10027604399999999 2023-11-01T04:16:33.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.344 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.344 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.344 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.016021+0000", 2023-11-01T04:16:33.344 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6206999999999999e-05 2023-11-01T04:16:33.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.345 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.345 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.019629+0000", 2023-11-01T04:16:33.345 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.003608076 2023-11-01T04:16:33.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.346 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.346 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.346 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.019721+0000", 2023-11-01T04:16:33.346 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.1978000000000006e-05 2023-11-01T04:16:33.346 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.346 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.347 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.347 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.3s3 51/31 ECSubWrite(tid=252, reqid=client.4559.0:396, at_version=51'17, trim_to=0'0, roll_forward_to=47'15))", 2023-11-01T04:16:33.347 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:31.287677+0000", 2023-11-01T04:16:33.347 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 1.879289368, 2023-11-01T04:16:33.348 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.14806313600000001, 2023-11-01T04:16:33.348 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.348 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.348 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.348 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.348 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.349 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.287677+0000", 2023-11-01T04:16:33.349 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.349 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.350 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.287677+0000", 2023-11-01T04:16:33.350 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.350 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.350 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.287678+0000", 2023-11-01T04:16:33.351 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.576e-06 2023-11-01T04:16:33.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.351 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.351 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.288323+0000", 2023-11-01T04:16:33.351 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00064454900000000001 2023-11-01T04:16:33.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.352 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.352 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.352 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.288346+0000", 2023-11-01T04:16:33.352 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2908000000000002e-05 2023-11-01T04:16:33.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.353 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.353 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.353 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.288357+0000", 2023-11-01T04:16:33.353 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1606e-05 2023-11-01T04:16:33.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.353 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.354 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.354 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.288465+0000", 2023-11-01T04:16:33.354 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000107484 2023-11-01T04:16:33.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.354 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.354 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.288476+0000", 2023-11-01T04:16:33.355 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1331999999999999e-05 2023-11-01T04:16:33.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.355 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.355 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.435711+0000", 2023-11-01T04:16:33.356 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.14723535500000001 2023-11-01T04:16:33.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.356 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.356 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.356 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.435740+0000", 2023-11-01T04:16:33.356 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8326e-05 2023-11-01T04:16:33.357 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.357 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.357 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.357 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.358 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.3s3 51/31 ECSubWrite(tid=253, reqid=client.4559.0:397, at_version=51'18, trim_to=0'0, roll_forward_to=47'15))", 2023-11-01T04:16:33.358 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:31.291875+0000", 2023-11-01T04:16:33.358 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 1.8750912319999999, 2023-11-01T04:16:33.358 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.14389136, 2023-11-01T04:16:33.358 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.359 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.359 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.359 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.359 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.359 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.291875+0000", 2023-11-01T04:16:33.360 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.360 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.360 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.360 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.291875+0000", 2023-11-01T04:16:33.360 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.361 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.361 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.291876+0000", 2023-11-01T04:16:33.361 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4499999999999996e-07 2023-11-01T04:16:33.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.362 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.362 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.292099+0000", 2023-11-01T04:16:33.362 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00022374600000000001 2023-11-01T04:16:33.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.363 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.363 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.292109+0000", 2023-11-01T04:16:33.363 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7489999999999993e-06 2023-11-01T04:16:33.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.363 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.364 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.292116+0000", 2023-11-01T04:16:33.364 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7939999999999997e-06 2023-11-01T04:16:33.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.364 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.364 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.292170+0000", 2023-11-01T04:16:33.365 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4256999999999999e-05 2023-11-01T04:16:33.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.365 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.365 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.365 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.292177+0000", 2023-11-01T04:16:33.365 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0339999999999999e-06 2023-11-01T04:16:33.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.366 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.366 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.435754+0000", 2023-11-01T04:16:33.366 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.143576919 2023-11-01T04:16:33.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.367 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.367 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.435766+0000", 2023-11-01T04:16:33.367 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2116000000000001e-05 2023-11-01T04:16:33.367 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.367 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.368 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.368 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.368 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.368 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.3s3 51/31 ECSubWrite(tid=254, reqid=client.4559.0:398, at_version=51'19, trim_to=0'0, roll_forward_to=47'15))", 2023-11-01T04:16:33.368 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:31.294962+0000", 2023-11-01T04:16:33.368 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 1.8720035940000002, 2023-11-01T04:16:33.369 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.14082186299999999, 2023-11-01T04:16:33.369 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.369 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.369 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.369 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.370 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.370 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.294962+0000", 2023-11-01T04:16:33.370 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.370 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.371 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.371 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.294962+0000", 2023-11-01T04:16:33.371 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.371 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.371 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.294963+0000", 2023-11-01T04:16:33.372 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1399999999999996e-07 2023-11-01T04:16:33.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.372 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.372 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.294984+0000", 2023-11-01T04:16:33.373 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1234999999999999e-05 2023-11-01T04:16:33.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.373 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.294996+0000", 2023-11-01T04:16:33.373 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1284000000000001e-05 2023-11-01T04:16:33.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.374 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.374 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.374 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.295003+0000", 2023-11-01T04:16:33.374 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8229999999999997e-06 2023-11-01T04:16:33.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.375 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.375 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.395175+0000", 2023-11-01T04:16:33.375 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10017290700000001 2023-11-01T04:16:33.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.376 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.376 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.395192+0000", 2023-11-01T04:16:33.376 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6881e-05 2023-11-01T04:16:33.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.376 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.376 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.377 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.435768+0000", 2023-11-01T04:16:33.377 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.040575437999999998 2023-11-01T04:16:33.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.377 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.377 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.435784+0000", 2023-11-01T04:16:33.377 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6580999999999999e-05 2023-11-01T04:16:33.378 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.378 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.378 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.378 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.378 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.379 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:33.380 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-01T04:16:33.380 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:33.378+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:33.481 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:33.481+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:33.494 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:33.495 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:16:33.495 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.495 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.9s4 53/31 ECSubWrite(tid=51, reqid=client.4559.0:406, at_version=53'41, trim_to=0'0, roll_forward_to=53'39))", 2023-11-01T04:16:33.495 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:33.418656+0000", 2023-11-01T04:16:33.495 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.077250067000000006, 2023-11-01T04:16:33.496 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.077268709000000005, 2023-11-01T04:16:33.496 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.496 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:33.496 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.496 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.496 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.497 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.418656+0000", 2023-11-01T04:16:33.497 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.497 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.497 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.497 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.498 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.418656+0000", 2023-11-01T04:16:33.498 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.498 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.498 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.498 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.498 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.418656+0000", 2023-11-01T04:16:33.499 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9100000000000003e-07 2023-11-01T04:16:33.499 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.499 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.499 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.499 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.418763+0000", 2023-11-01T04:16:33.500 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000106166 2023-11-01T04:16:33.500 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.500 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.500 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.500 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.418776+0000", 2023-11-01T04:16:33.501 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.311e-05 2023-11-01T04:16:33.501 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.501 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.501 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.501 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.418787+0000", 2023-11-01T04:16:33.502 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1068e-05 2023-11-01T04:16:33.502 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.502 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.502 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.502 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.503 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-11-01T04:16:33.503 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 1 2023-11-01T04:16:33.503 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:33.508 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-01T04:16:33.582 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:33.581+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:33.627 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:33.628 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:16:33.628 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2023-11-01T04:16:33.628 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2023-11-01T04:16:33.628 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:33.642 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-01T04:16:33.683 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:33.681+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:33.762 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:33.762 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2023-11-01T04:16:33.762 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2023-11-01T04:16:33.762 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:16:33.763 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.763 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.es3 36/31 ECSubWrite(tid=12, reqid=client.4559.0:133, at_version=36'7, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:33.763 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.169895+0000", 2023-11-01T04:16:33.763 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 18.590664632999999, 2023-11-01T04:16:33.763 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103016208, 2023-11-01T04:16:33.764 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.764 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.764 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.764 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.764 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.765 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.169895+0000", 2023-11-01T04:16:33.765 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.765 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.765 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.765 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.766 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.169895+0000", 2023-11-01T04:16:33.766 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.766 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.766 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.766 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.767 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.169897+0000", 2023-11-01T04:16:33.767 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.466e-06 2023-11-01T04:16:33.767 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.767 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.767 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.768 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.170419+0000", 2023-11-01T04:16:33.768 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00052250599999999995 2023-11-01T04:16:33.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.768 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.769 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.769 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.170457+0000", 2023-11-01T04:16:33.769 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.7945999999999999e-05 2023-11-01T04:16:33.769 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.769 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.770 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.770 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.170474+0000", 2023-11-01T04:16:33.770 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7045999999999999e-05 2023-11-01T04:16:33.770 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.770 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.771 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.771 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.270728+0000", 2023-11-01T04:16:33.771 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100254038 2023-11-01T04:16:33.771 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.771 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.772 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.772 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.270744+0000", 2023-11-01T04:16:33.772 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.507e-05 2023-11-01T04:16:33.772 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.772 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.773 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.773 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.272841+0000", 2023-11-01T04:16:33.773 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0020970310000000001 2023-11-01T04:16:33.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.773 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.774 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.774 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.272912+0000", 2023-11-01T04:16:33.774 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1105000000000004e-05 2023-11-01T04:16:33.774 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.774 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.775 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.775 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.775 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.775 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.as1 36/31 ECSubWrite(tid=24, reqid=client.4559.0:140, at_version=36'4, trim_to=0'0, roll_forward_to=34'2))", 2023-11-01T04:16:33.775 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.199076+0000", 2023-11-01T04:16:33.776 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 18.561483976000002, 2023-11-01T04:16:33.776 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20133796800000001, 2023-11-01T04:16:33.776 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.776 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.776 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.777 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.777 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.777 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199076+0000", 2023-11-01T04:16:33.777 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.777 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.777 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.778 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.778 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199076+0000", 2023-11-01T04:16:33.778 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.778 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.779 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.779 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199078+0000", 2023-11-01T04:16:33.779 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.623e-06 2023-11-01T04:16:33.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.780 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.780 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.780 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199121+0000", 2023-11-01T04:16:33.780 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3727999999999999e-05 2023-11-01T04:16:33.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.781 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.781 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.781 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199278+0000", 2023-11-01T04:16:33.781 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015602899999999999 2023-11-01T04:16:33.781 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.781 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.782 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.782 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299400+0000", 2023-11-01T04:16:33.782 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100122192 2023-11-01T04:16:33.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.782 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.783 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.783 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399577+0000", 2023-11-01T04:16:33.783 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10017696500000001 2023-11-01T04:16:33.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.783 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.784 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.784 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399597+0000", 2023-11-01T04:16:33.784 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0213000000000001e-05 2023-11-01T04:16:33.784 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.784 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.784 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.785 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.400376+0000", 2023-11-01T04:16:33.785 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00077926799999999995 2023-11-01T04:16:33.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.785 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.785 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.786 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.400414+0000", 2023-11-01T04:16:33.786 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.7950000000000001e-05 2023-11-01T04:16:33.786 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.786 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.786 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.787 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.787 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.0s2 36/31 ECSubWrite(tid=6, reqid=client.4559.0:137, at_version=0'0, trim_to=0'0, roll_forward_to=36'4))", 2023-11-01T04:16:33.787 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.609108+0000", 2023-11-01T04:16:33.787 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 18.151451987000002, 2023-11-01T04:16:33.787 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20084221199999999, 2023-11-01T04:16:33.788 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.788 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.788 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.788 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.788 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.789 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.609108+0000", 2023-11-01T04:16:33.789 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.789 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.789 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.789 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.790 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.609108+0000", 2023-11-01T04:16:33.790 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.790 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.790 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.791 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.609109+0000", 2023-11-01T04:16:33.791 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.257e-06 2023-11-01T04:16:33.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.791 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.791 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.792 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.609125+0000", 2023-11-01T04:16:33.792 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5305e-05 2023-11-01T04:16:33.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.792 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.793 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.609143+0000", 2023-11-01T04:16:33.793 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8746e-05 2023-11-01T04:16:33.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.793 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.793 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.793 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709231+0000", 2023-11-01T04:16:33.794 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100087153 2023-11-01T04:16:33.794 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.794 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.794 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.794 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709326+0000", 2023-11-01T04:16:33.795 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5130000000000006e-05 2023-11-01T04:16:33.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.795 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.795 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.795 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709341+0000", 2023-11-01T04:16:33.796 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5699999999999999e-05 2023-11-01T04:16:33.796 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.796 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.796 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.796 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.809924+0000", 2023-11-01T04:16:33.797 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10058260099999999 2023-11-01T04:16:33.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.797 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.797 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.797 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.809950+0000", 2023-11-01T04:16:33.797 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6319999999999999e-05 2023-11-01T04:16:33.798 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.798 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.798 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.798 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.798 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.799 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds1 36/31 ECSubWrite(tid=30, reqid=client.4559.0:143, at_version=36'14, trim_to=0'0, roll_forward_to=36'13))", 2023-11-01T04:16:33.799 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.668264+0000", 2023-11-01T04:16:33.799 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 18.092295986, 2023-11-01T04:16:33.799 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20128238700000001, 2023-11-01T04:16:33.799 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.800 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.800 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.800 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.800 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.800 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.668264+0000", 2023-11-01T04:16:33.801 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.801 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.801 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.801 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.801 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.668264+0000", 2023-11-01T04:16:33.801 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.802 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.802 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.802 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.802 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.668265+0000", 2023-11-01T04:16:33.802 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0699999999999996e-07 2023-11-01T04:16:33.803 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.803 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.803 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.803 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.668318+0000", 2023-11-01T04:16:33.803 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3316000000000002e-05 2023-11-01T04:16:33.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.804 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.804 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.804 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.668335+0000", 2023-11-01T04:16:33.805 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6935999999999998e-05 2023-11-01T04:16:33.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.805 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.805 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.805 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.768415+0000", 2023-11-01T04:16:33.806 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100080154 2023-11-01T04:16:33.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.806 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.806 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.806 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.768508+0000", 2023-11-01T04:16:33.807 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2936000000000006e-05 2023-11-01T04:16:33.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.807 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.807 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.807 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.768521+0000", 2023-11-01T04:16:33.808 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2683999999999999e-05 2023-11-01T04:16:33.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.808 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.808 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.808 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.869474+0000", 2023-11-01T04:16:33.808 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100952606 2023-11-01T04:16:33.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.809 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.809 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.809 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.869547+0000", 2023-11-01T04:16:33.809 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2947999999999993e-05 2023-11-01T04:16:33.810 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.810 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.810 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.810 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.811 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds1 36/31 ECSubWrite(tid=31, reqid=client.4559.0:144, at_version=36'15, trim_to=0'0, roll_forward_to=36'13))", 2023-11-01T04:16:33.811 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.768458+0000", 2023-11-01T04:16:33.811 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 17.992102246999998, 2023-11-01T04:16:33.811 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.102118419, 2023-11-01T04:16:33.811 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.811 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.812 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.812 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.812 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.812 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.768458+0000", 2023-11-01T04:16:33.812 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.813 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.813 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.813 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.768458+0000", 2023-11-01T04:16:33.813 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.814 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.814 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.814 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.768459+0000", 2023-11-01T04:16:33.814 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5900000000000005e-07 2023-11-01T04:16:33.814 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.815 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.815 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.815 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.768481+0000", 2023-11-01T04:16:33.815 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2235999999999999e-05 2023-11-01T04:16:33.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.816 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.816 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.816 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.768499+0000", 2023-11-01T04:16:33.816 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8332e-05 2023-11-01T04:16:33.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.816 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.817 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.817 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.768511+0000", 2023-11-01T04:16:33.817 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.201e-05 2023-11-01T04:16:33.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.817 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.818 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.818 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.869108+0000", 2023-11-01T04:16:33.818 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10059673500000001 2023-11-01T04:16:33.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.818 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.818 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.819 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.869126+0000", 2023-11-01T04:16:33.819 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7452000000000002e-05 2023-11-01T04:16:33.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.819 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.819 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.820 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.870512+0000", 2023-11-01T04:16:33.820 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0013862939999999999 2023-11-01T04:16:33.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.820 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.820 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.821 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.870576+0000", 2023-11-01T04:16:33.821 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4400999999999995e-05 2023-11-01T04:16:33.821 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.821 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.821 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.822 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.822 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.ds1 36/31 ECSubRead(tid=33, to_read={3:b31c2ca6:::smithi19039307-48:head=0,4096,0}, subchunks={3:b31c2ca6:::smithi19039307-48:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:33.822 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.872270+0000", 2023-11-01T04:16:33.822 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 17.888290443999999, 2023-11-01T04:16:33.822 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.199929154, 2023-11-01T04:16:33.823 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.823 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.823 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.823 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.823 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.824 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.872270+0000", 2023-11-01T04:16:33.824 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.824 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.824 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.824 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.872270+0000", 2023-11-01T04:16:33.825 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.825 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.825 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.825 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.872271+0000", 2023-11-01T04:16:33.825 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.517e-06 2023-11-01T04:16:33.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.826 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.826 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.826 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.872291+0000", 2023-11-01T04:16:33.826 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0060000000000001e-05 2023-11-01T04:16:33.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.827 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.827 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.827 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.872331+0000", 2023-11-01T04:16:33.827 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0234999999999997e-05 2023-11-01T04:16:33.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.828 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.828 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.828 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.872362+0000", 2023-11-01T04:16:33.828 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0315999999999999e-05 2023-11-01T04:16:33.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.828 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.829 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.829 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.071857+0000", 2023-11-01T04:16:33.829 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19949471299999999 2023-11-01T04:16:33.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.829 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.830 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.830 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.072199+0000", 2023-11-01T04:16:33.830 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00034231300000000001 2023-11-01T04:16:33.830 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.830 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.831 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.831 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.831 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.es3 37/31 ECSubWrite(tid=21, reqid=client.4559.0:186, at_version=37'14, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:16:33.831 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.967923+0000", 2023-11-01T04:16:33.831 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 16.792637343999999, 2023-11-01T04:16:33.832 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10252109500000001, 2023-11-01T04:16:33.832 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.832 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.832 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.832 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.833 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.833 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967923+0000", 2023-11-01T04:16:33.833 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.833 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.833 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.833 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.834 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967923+0000", 2023-11-01T04:16:33.834 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.834 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.834 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.834 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.835 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967924+0000", 2023-11-01T04:16:33.835 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3600000000000002e-07 2023-11-01T04:16:33.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.835 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.835 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.835 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967943+0000", 2023-11-01T04:16:33.836 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9168999999999999e-05 2023-11-01T04:16:33.836 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.836 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.836 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.836 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967958+0000", 2023-11-01T04:16:33.836 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4802e-05 2023-11-01T04:16:33.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.837 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.837 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.837 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967965+0000", 2023-11-01T04:16:33.837 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.7130000000000005e-06 2023-11-01T04:16:33.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.838 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.838 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.838 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.065569+0000", 2023-11-01T04:16:33.838 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.097603500999999995 2023-11-01T04:16:33.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.839 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.839 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.839 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.065584+0000", 2023-11-01T04:16:33.839 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5438e-05 2023-11-01T04:16:33.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.840 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.840 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.840 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.070416+0000", 2023-11-01T04:16:33.840 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0048317789999999996 2023-11-01T04:16:33.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.841 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.841 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.841 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.070444+0000", 2023-11-01T04:16:33.841 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7857000000000001e-05 2023-11-01T04:16:33.841 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.841 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.842 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.842 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.842 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds1 36/31 ECSubWrite(tid=43, reqid=client.4559.0:169, at_version=0'0, trim_to=0'0, roll_forward_to=36'17))", 2023-11-01T04:16:33.842 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:17.027385+0000", 2023-11-01T04:16:33.843 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 16.733174675000001, 2023-11-01T04:16:33.843 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.102017624, 2023-11-01T04:16:33.843 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.843 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.843 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.843 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.844 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.844 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.027385+0000", 2023-11-01T04:16:33.844 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.844 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.844 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.845 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.845 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.027385+0000", 2023-11-01T04:16:33.845 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.845 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.845 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.846 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.027389+0000", 2023-11-01T04:16:33.846 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6399999999999999e-06 2023-11-01T04:16:33.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.846 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.846 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.846 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.027424+0000", 2023-11-01T04:16:33.847 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4479e-05 2023-11-01T04:16:33.847 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.847 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.847 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.847 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.027470+0000", 2023-11-01T04:16:33.848 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6514999999999998e-05 2023-11-01T04:16:33.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.848 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.848 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.848 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.127576+0000", 2023-11-01T04:16:33.848 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100105613 2023-11-01T04:16:33.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.849 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.849 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.849 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.127834+0000", 2023-11-01T04:16:33.849 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00025842299999999998 2023-11-01T04:16:33.850 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.850 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.850 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.850 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.127857+0000", 2023-11-01T04:16:33.850 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2739e-05 2023-11-01T04:16:33.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.851 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.851 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.851 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.129330+0000", 2023-11-01T04:16:33.851 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0014726940000000001 2023-11-01T04:16:33.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.852 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.852 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.852 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.129403+0000", 2023-11-01T04:16:33.852 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3521000000000003e-05 2023-11-01T04:16:33.852 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.852 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.853 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.853 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.853 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.853 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds1 37/31 ECSubWrite(tid=44, reqid=client.4559.0:189, at_version=37'19, trim_to=0'0, roll_forward_to=36'17))", 2023-11-01T04:16:33.853 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:17.267187+0000", 2023-11-01T04:16:33.854 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 16.493372751999999, 2023-11-01T04:16:33.854 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.102825324, 2023-11-01T04:16:33.854 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.854 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.854 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.854 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.855 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.855 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267187+0000", 2023-11-01T04:16:33.855 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.855 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.855 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.856 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267187+0000", 2023-11-01T04:16:33.856 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.856 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.856 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.856 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.856 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267189+0000", 2023-11-01T04:16:33.857 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.807e-06 2023-11-01T04:16:33.857 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.857 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.857 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.857 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267255+0000", 2023-11-01T04:16:33.858 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6070999999999998e-05 2023-11-01T04:16:33.858 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.858 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.858 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.858 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267294+0000", 2023-11-01T04:16:33.858 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8716e-05 2023-11-01T04:16:33.859 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.859 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.859 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.859 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267318+0000", 2023-11-01T04:16:33.859 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3598999999999999e-05 2023-11-01T04:16:33.860 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.860 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.860 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.860 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.367569+0000", 2023-11-01T04:16:33.860 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100251039 2023-11-01T04:16:33.861 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.861 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.861 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.861 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.367583+0000", 2023-11-01T04:16:33.861 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4473e-05 2023-11-01T04:16:33.861 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.862 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.862 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.862 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.369922+0000", 2023-11-01T04:16:33.862 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0023389890000000001 2023-11-01T04:16:33.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.862 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.863 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.863 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.370013+0000", 2023-11-01T04:16:33.863 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0630000000000005e-05 2023-11-01T04:16:33.863 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.863 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.863 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.864 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.864 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.es3 37/31 ECSubWrite(tid=24, reqid=client.4559.0:194, at_version=37'17, trim_to=0'0, roll_forward_to=37'15))", 2023-11-01T04:16:33.864 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:17.480035+0000", 2023-11-01T04:16:33.864 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 16.280525427000001, 2023-11-01T04:16:33.865 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10411511599999999, 2023-11-01T04:16:33.865 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.865 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.865 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.865 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.865 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.866 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.480035+0000", 2023-11-01T04:16:33.866 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.866 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.866 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.866 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.866 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.480035+0000", 2023-11-01T04:16:33.867 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.867 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.867 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.867 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.867 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.480037+0000", 2023-11-01T04:16:33.867 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9300000000000002e-06 2023-11-01T04:16:33.868 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.868 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.868 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.868 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.480565+0000", 2023-11-01T04:16:33.868 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00052879800000000003 2023-11-01T04:16:33.869 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.869 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.869 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.869 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.480604+0000", 2023-11-01T04:16:33.869 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8325000000000003e-05 2023-11-01T04:16:33.869 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.870 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.870 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.870 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.480626+0000", 2023-11-01T04:16:33.870 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2646e-05 2023-11-01T04:16:33.870 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.871 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.871 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.871 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.580869+0000", 2023-11-01T04:16:33.871 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100242556 2023-11-01T04:16:33.871 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.871 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.872 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.872 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.580883+0000", 2023-11-01T04:16:33.872 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4273e-05 2023-11-01T04:16:33.872 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.872 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.873 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.873 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.584055+0000", 2023-11-01T04:16:33.873 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0031720569999999998 2023-11-01T04:16:33.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.873 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.873 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.874 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.584150+0000", 2023-11-01T04:16:33.874 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4530999999999993e-05 2023-11-01T04:16:33.874 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.874 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.874 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.874 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.875 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.875 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s3 37/31 ECSubWrite(tid=209, reqid=client.4559.0:207, at_version=0'0, trim_to=0'0, roll_forward_to=37'21))", 2023-11-01T04:16:33.875 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:17.586347+0000", 2023-11-01T04:16:33.875 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 16.174212738000001, 2023-11-01T04:16:33.875 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10204546, 2023-11-01T04:16:33.875 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.876 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.876 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.876 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.876 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.876 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.586347+0000", 2023-11-01T04:16:33.876 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.877 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.877 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.877 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.877 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.586347+0000", 2023-11-01T04:16:33.877 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.877 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.878 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.878 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.878 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.586349+0000", 2023-11-01T04:16:33.878 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.42e-06 2023-11-01T04:16:33.878 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.879 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.879 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.879 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.586366+0000", 2023-11-01T04:16:33.879 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7558999999999999e-05 2023-11-01T04:16:33.879 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.880 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.880 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.880 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.586394+0000", 2023-11-01T04:16:33.880 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7172999999999999e-05 2023-11-01T04:16:33.880 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.880 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.881 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.881 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.686549+0000", 2023-11-01T04:16:33.881 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100155415 2023-11-01T04:16:33.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.881 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.881 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.882 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.686693+0000", 2023-11-01T04:16:33.882 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014377899999999999 2023-11-01T04:16:33.882 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.882 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.882 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.883 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.686721+0000", 2023-11-01T04:16:33.883 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7829000000000001e-05 2023-11-01T04:16:33.883 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.883 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.883 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.883 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.688310+0000", 2023-11-01T04:16:33.884 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0015892840000000001 2023-11-01T04:16:33.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.884 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.884 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.884 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.688393+0000", 2023-11-01T04:16:33.884 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3001e-05 2023-11-01T04:16:33.885 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.885 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.885 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.885 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.885 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.886 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds1 41/31 ECSubWrite(tid=55, reqid=client.4559.0:266, at_version=41'21, trim_to=0'0, roll_forward_to=37'19))", 2023-11-01T04:16:33.886 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.123059+0000", 2023-11-01T04:16:33.886 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 12.637501375999999, 2023-11-01T04:16:33.886 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103087974, 2023-11-01T04:16:33.886 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.886 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.887 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.887 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.887 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.887 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123059+0000", 2023-11-01T04:16:33.887 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.887 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.888 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.888 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.888 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123059+0000", 2023-11-01T04:16:33.888 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.888 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.889 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.889 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.889 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123061+0000", 2023-11-01T04:16:33.889 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8870000000000001e-06 2023-11-01T04:16:33.889 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.889 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.890 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.890 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123341+0000", 2023-11-01T04:16:33.890 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00028086000000000002 2023-11-01T04:16:33.890 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.890 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.891 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.891 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123374+0000", 2023-11-01T04:16:33.891 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2852000000000003e-05 2023-11-01T04:16:33.891 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.891 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.891 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.892 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.223454+0000", 2023-11-01T04:16:33.892 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100079638 2023-11-01T04:16:33.892 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.892 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.892 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.892 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.223532+0000", 2023-11-01T04:16:33.893 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.7715999999999999e-05 2023-11-01T04:16:33.893 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.893 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.893 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.893 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.223541+0000", 2023-11-01T04:16:33.893 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8550000000000008e-06 2023-11-01T04:16:33.894 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.894 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.894 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.894 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.226096+0000", 2023-11-01T04:16:33.894 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002555505 2023-11-01T04:16:33.895 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.895 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.895 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.895 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.226147+0000", 2023-11-01T04:16:33.895 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0661000000000002e-05 2023-11-01T04:16:33.895 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.896 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.896 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.896 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.896 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/31 ECSubWrite(tid=236, reqid=client.4559.0:330, at_version=46'36, trim_to=0'0, roll_forward_to=45'33))", 2023-11-01T04:16:33.897 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.282825+0000", 2023-11-01T04:16:33.897 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 7.4777351489999999, 2023-11-01T04:16:33.897 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20525125999999999, 2023-11-01T04:16:33.897 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.897 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.897 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.898 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.898 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.898 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282825+0000", 2023-11-01T04:16:33.898 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.898 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.898 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.899 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.899 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282825+0000", 2023-11-01T04:16:33.899 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.899 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.900 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.900 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.900 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282827+0000", 2023-11-01T04:16:33.900 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2800000000000002e-06 2023-11-01T04:16:33.900 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.901 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.901 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.901 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282955+0000", 2023-11-01T04:16:33.901 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012810400000000001 2023-11-01T04:16:33.901 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.901 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.902 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.902 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.283001+0000", 2023-11-01T04:16:33.902 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5411999999999998e-05 2023-11-01T04:16:33.902 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.902 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.902 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.903 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383146+0000", 2023-11-01T04:16:33.903 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100144996 2023-11-01T04:16:33.903 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.903 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.903 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.903 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383332+0000", 2023-11-01T04:16:33.904 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00018672099999999999 2023-11-01T04:16:33.904 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.904 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.904 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.904 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383371+0000", 2023-11-01T04:16:33.905 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8532999999999999e-05 2023-11-01T04:16:33.905 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.905 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.905 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.905 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.488032+0000", 2023-11-01T04:16:33.905 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104661192 2023-11-01T04:16:33.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.906 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.906 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.906 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.488076+0000", 2023-11-01T04:16:33.906 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4022e-05 2023-11-01T04:16:33.906 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.907 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.907 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.907 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.907 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.907 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/31 ECSubWrite(tid=237, reqid=client.4559.0:331, at_version=46'37, trim_to=0'0, roll_forward_to=45'33))", 2023-11-01T04:16:33.908 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.383221+0000", 2023-11-01T04:16:33.908 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 7.3773392940000004, 2023-11-01T04:16:33.908 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10489668100000001, 2023-11-01T04:16:33.908 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.908 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.908 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.909 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.909 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.909 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383221+0000", 2023-11-01T04:16:33.909 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.909 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.910 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.910 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.910 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383221+0000", 2023-11-01T04:16:33.910 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.910 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.911 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.911 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.911 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383222+0000", 2023-11-01T04:16:33.911 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.545e-06 2023-11-01T04:16:33.911 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.911 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.912 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.912 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383256+0000", 2023-11-01T04:16:33.912 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4094999999999998e-05 2023-11-01T04:16:33.912 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.912 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.913 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.913 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383273+0000", 2023-11-01T04:16:33.913 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6722999999999999e-05 2023-11-01T04:16:33.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.913 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.913 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.914 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383285+0000", 2023-11-01T04:16:33.914 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1728e-05 2023-11-01T04:16:33.914 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.914 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.914 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.914 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.384624+0000", 2023-11-01T04:16:33.915 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001339086 2023-11-01T04:16:33.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.915 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.915 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.915 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.384639+0000", 2023-11-01T04:16:33.915 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5328e-05 2023-11-01T04:16:33.916 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.916 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.916 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.916 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.488085+0000", 2023-11-01T04:16:33.916 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103445437 2023-11-01T04:16:33.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.917 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.917 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.917 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.488117+0000", 2023-11-01T04:16:33.917 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2738999999999999e-05 2023-11-01T04:16:33.917 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.918 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.918 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.918 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.918 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.918 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.2s3 46/31 ECSubRead(tid=238, to_read={3:4a13469b:::smithi19039307-24:head=0,561152,0}, subchunks={3:4a13469b:::smithi19039307-24:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:33.918 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.383307+0000", 2023-11-01T04:16:33.919 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 7.377252865, 2023-11-01T04:16:33.919 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104403442, 2023-11-01T04:16:33.919 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.919 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.919 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.920 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.920 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.920 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383307+0000", 2023-11-01T04:16:33.920 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.920 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.921 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.921 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.921 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383307+0000", 2023-11-01T04:16:33.921 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.921 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.921 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.922 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.922 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383308+0000", 2023-11-01T04:16:33.922 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6699999999999999e-07 2023-11-01T04:16:33.922 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.922 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.923 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.923 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383317+0000", 2023-11-01T04:16:33.923 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9959999999999999e-06 2023-11-01T04:16:33.923 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.923 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.923 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.924 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383331+0000", 2023-11-01T04:16:33.924 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4289e-05 2023-11-01T04:16:33.924 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.924 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.924 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.924 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383343+0000", 2023-11-01T04:16:33.925 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1986e-05 2023-11-01T04:16:33.925 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.925 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.925 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.925 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.484996+0000", 2023-11-01T04:16:33.925 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101652828 2023-11-01T04:16:33.926 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.926 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.926 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.926 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.487711+0000", 2023-11-01T04:16:33.926 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0027148760000000002 2023-11-01T04:16:33.926 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.927 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.927 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.927 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.927 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.2s3 46/31 ECSubRead(tid=239, to_read={3:4a13469b:::smithi19039307-24:head=0,561152,0}, subchunks={3:4a13469b:::smithi19039307-24:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:33.928 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.383384+0000", 2023-11-01T04:16:33.928 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 7.3771761299999996, 2023-11-01T04:16:33.928 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104738817, 2023-11-01T04:16:33.928 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.928 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.928 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.929 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.929 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.929 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383384+0000", 2023-11-01T04:16:33.929 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.929 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.930 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.930 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.930 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383384+0000", 2023-11-01T04:16:33.930 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.931 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.931 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.931 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383384+0000", 2023-11-01T04:16:33.931 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9000000000000002e-07 2023-11-01T04:16:33.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.931 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.932 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.932 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383391+0000", 2023-11-01T04:16:33.932 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9870000000000002e-06 2023-11-01T04:16:33.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.932 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.932 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.933 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383406+0000", 2023-11-01T04:16:33.933 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4270000000000001e-05 2023-11-01T04:16:33.933 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.933 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.933 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.934 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383420+0000", 2023-11-01T04:16:33.934 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3890000000000001e-05 2023-11-01T04:16:33.934 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.934 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.934 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.934 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.487782+0000", 2023-11-01T04:16:33.935 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10436245399999999 2023-11-01T04:16:33.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.935 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.935 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.935 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.488123+0000", 2023-11-01T04:16:33.936 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00034082600000000002 2023-11-01T04:16:33.936 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.936 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.936 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.936 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.937 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.2s3 46/31 ECSubRead(tid=240, to_read={3:4a13469b:::smithi19039307-24:head=0,561152,0}, subchunks={3:4a13469b:::smithi19039307-24:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:33.937 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.383437+0000", 2023-11-01T04:16:33.937 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 7.3771231239999997, 2023-11-01T04:16:33.937 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10541188, 2023-11-01T04:16:33.937 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.937 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.938 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.938 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.938 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.938 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383437+0000", 2023-11-01T04:16:33.938 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.939 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.939 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.939 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383437+0000", 2023-11-01T04:16:33.939 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.940 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.940 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.940 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383437+0000", 2023-11-01T04:16:33.940 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5999999999999999e-07 2023-11-01T04:16:33.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.941 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.941 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.941 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383445+0000", 2023-11-01T04:16:33.941 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6299999999999998e-06 2023-11-01T04:16:33.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.941 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.942 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.942 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383457+0000", 2023-11-01T04:16:33.942 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1555e-05 2023-11-01T04:16:33.942 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.942 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.942 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.943 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383476+0000", 2023-11-01T04:16:33.943 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9397000000000001e-05 2023-11-01T04:16:33.943 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.943 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.943 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.944 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.488221+0000", 2023-11-01T04:16:33.944 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104745329 2023-11-01T04:16:33.944 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.944 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.944 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.944 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.488849+0000", 2023-11-01T04:16:33.945 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000627609 2023-11-01T04:16:33.945 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.945 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.945 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.945 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.945 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.946 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.1s1 47/31 ECSubWrite(tid=43, reqid=client.4559.0:355, at_version=47'23, trim_to=0'0, roll_forward_to=46'20))", 2023-11-01T04:16:33.946 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.484592+0000", 2023-11-01T04:16:33.946 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.2759677030000001, 2023-11-01T04:16:33.946 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.102474474, 2023-11-01T04:16:33.946 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.946 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.947 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.947 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.947 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.947 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.484592+0000", 2023-11-01T04:16:33.947 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.947 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.948 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.948 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.948 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.484592+0000", 2023-11-01T04:16:33.948 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.948 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.949 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.949 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.949 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.484594+0000", 2023-11-01T04:16:33.949 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3260000000000001e-06 2023-11-01T04:16:33.949 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.950 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.950 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.950 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.484755+0000", 2023-11-01T04:16:33.950 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016139 2023-11-01T04:16:33.950 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.951 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.951 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.951 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.484779+0000", 2023-11-01T04:16:33.951 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3419000000000001e-05 2023-11-01T04:16:33.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.951 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.952 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.952 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.584938+0000", 2023-11-01T04:16:33.952 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100159436 2023-11-01T04:16:33.952 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.952 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.952 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.953 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.585078+0000", 2023-11-01T04:16:33.953 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000140414 2023-11-01T04:16:33.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.953 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.953 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.953 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.585105+0000", 2023-11-01T04:16:33.954 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6533000000000002e-05 2023-11-01T04:16:33.954 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.954 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.954 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.954 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.586976+0000", 2023-11-01T04:16:33.955 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001871501 2023-11-01T04:16:33.955 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.955 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.955 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.955 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.587067+0000", 2023-11-01T04:16:33.955 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0455e-05 2023-11-01T04:16:33.956 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.956 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.956 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.956 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.956 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.956 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.9s4 51/31 ECSubWrite(tid=47, reqid=client.4559.0:394, at_version=0'0, trim_to=0'0, roll_forward_to=51'37))", 2023-11-01T04:16:33.957 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:31.784104+0000", 2023-11-01T04:16:33.957 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 1.9764558999999999, 2023-11-01T04:16:33.957 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101899192, 2023-11-01T04:16:33.957 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.957 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.958 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.958 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.958 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.958 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.784104+0000", 2023-11-01T04:16:33.958 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.958 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.959 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.959 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.959 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.784104+0000", 2023-11-01T04:16:33.959 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.960 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.960 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.960 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.784106+0000", 2023-11-01T04:16:33.960 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8139999999999999e-06 2023-11-01T04:16:33.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.961 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.961 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.961 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.784127+0000", 2023-11-01T04:16:33.961 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0990999999999999e-05 2023-11-01T04:16:33.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.961 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.962 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.962 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.784149+0000", 2023-11-01T04:16:33.962 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2022e-05 2023-11-01T04:16:33.962 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.962 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.963 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.963 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.884291+0000", 2023-11-01T04:16:33.963 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100142446 2023-11-01T04:16:33.963 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.963 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.963 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.964 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.884475+0000", 2023-11-01T04:16:33.964 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00018394900000000001 2023-11-01T04:16:33.964 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.964 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.964 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.965 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.884494+0000", 2023-11-01T04:16:33.965 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8393000000000002e-05 2023-11-01T04:16:33.965 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.965 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.965 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.965 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.885915+0000", 2023-11-01T04:16:33.966 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0014207950000000001 2023-11-01T04:16:33.966 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.966 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.966 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.966 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:31.886003+0000", 2023-11-01T04:16:33.966 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8782000000000005e-05 2023-11-01T04:16:33.967 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.967 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.967 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.967 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.967 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.967 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.9s4 53/31 ECSubWrite(tid=49, reqid=client.4559.0:408, at_version=53'39, trim_to=0'0, roll_forward_to=51'37))", 2023-11-01T04:16:33.968 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:33.209188+0000", 2023-11-01T04:16:33.968 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.55137219800000004, 2023-11-01T04:16:33.968 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10547720200000001, 2023-11-01T04:16:33.968 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:33.968 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:33.969 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:33.969 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.969 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:33.969 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.209188+0000", 2023-11-01T04:16:33.969 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.970 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.970 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.970 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:33.970 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.209188+0000", 2023-11-01T04:16:33.970 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:33.970 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.971 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.971 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:33.971 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.209188+0000", 2023-11-01T04:16:33.971 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9000000000000001e-07 2023-11-01T04:16:33.971 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.971 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.972 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:33.972 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.209540+0000", 2023-11-01T04:16:33.972 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00035237099999999998 2023-11-01T04:16:33.972 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.972 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.972 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:33.973 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.209559+0000", 2023-11-01T04:16:33.973 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8059000000000001e-05 2023-11-01T04:16:33.973 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.973 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.973 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:33.974 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.309662+0000", 2023-11-01T04:16:33.974 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100103894 2023-11-01T04:16:33.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.974 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.974 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:33.974 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.310907+0000", 2023-11-01T04:16:33.975 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0012448769999999999 2023-11-01T04:16:33.975 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.975 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.975 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:33.975 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.310919+0000", 2023-11-01T04:16:33.975 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1763000000000001e-05 2023-11-01T04:16:33.976 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.976 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.976 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:33.976 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.314538+0000", 2023-11-01T04:16:33.976 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00361854 2023-11-01T04:16:33.976 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:33.977 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:33.977 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:33.977 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.314665+0000", 2023-11-01T04:16:33.977 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000127508 2023-11-01T04:16:33.977 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.978 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.978 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.978 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:33.978 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:33.978 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:33.979 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-01T04:16:33.980 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-01T04:16:33.981 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:33.980+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:34.081 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:34.077+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:34.097 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.14 to 1 (10000) 2023-11-01T04:16:34.106 INFO:tasks.rados.rados.0.smithi190.stdout:164: done (4 left) 2023-11-01T04:16:34.107 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 25 current snap is 13 2023-11-01T04:16:34.107 INFO:tasks.rados.rados.0.smithi190.stdout:165: seq_num 76 ranges {1310720=507904,1818624=507904,2326528=262144} 2023-11-01T04:16:34.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-11-01T04:16:34.134 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2023-11-01T04:16:34.137 INFO:tasks.rados.rados.0.smithi190.stdout:165: writing smithi19039307-25 from 1310720 to 1818624 tid 1 2023-11-01T04:16:34.146 INFO:tasks.rados.rados.0.smithi190.stdout:165: writing smithi19039307-25 from 1818624 to 2326528 tid 2 2023-11-01T04:16:34.150 INFO:tasks.rados.rados.0.smithi190.stdout:165: writing smithi19039307-25 from 2326528 to 2588672 tid 3 2023-11-01T04:16:34.150 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 35 current snap is 13 2023-11-01T04:16:34.151 INFO:tasks.rados.rados.0.smithi190.stdout:166: seq_num 77 ranges {1785856=524288,2310144=524288,2834432=245760} 2023-11-01T04:16:34.168 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:16:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.6s3 54/31 ECSubWrite(tid=51, reqid=osd.14.0:51, at_version=54'8, trim_to=0'0, roll_forward_to=47'6))", 2023-11-01T04:16:34.168 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:34.106047+0000", 2023-11-01T04:16:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.062973800999999996, 2023-11-01T04:16:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.062999199000000006, 2023-11-01T04:16:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "no flag points reached", 2023-11-01T04:16:34.169 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:34.106047+0000", 2023-11-01T04:16:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:34.170 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:34.106047+0000", 2023-11-01T04:16:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:34.171 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:34.172 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:34.106048+0000", 2023-11-01T04:16:34.172 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.16e-07 2023-11-01T04:16:34.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:34.172 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:34.172 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:34.172 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:34.106100+0000", 2023-11-01T04:16:34.173 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2302000000000002e-05 2023-11-01T04:16:34.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:34.173 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:34.173 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:34.173 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:34.106140+0000", 2023-11-01T04:16:34.173 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9449000000000001e-05 2023-11-01T04:16:34.174 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:34.174 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:34.174 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:34.174 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:34.174 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-11-01T04:16:34.174 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 1 2023-11-01T04:16:34.175 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:34.175 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:34.176 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-11-01T04:16:34.179 INFO:tasks.rados.rados.0.smithi190.stdout:166: writing smithi19039307-35 from 1785856 to 2310144 tid 1 2023-11-01T04:16:34.181 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:34.179+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:34.183 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-01T04:16:34.184 INFO:tasks.rados.rados.0.smithi190.stdout:166: writing smithi19039307-35 from 2310144 to 2834432 tid 2 2023-11-01T04:16:34.186 INFO:tasks.rados.rados.0.smithi190.stdout:166: writing smithi19039307-35 from 2834432 to 3080192 tid 3 2023-11-01T04:16:34.186 INFO:tasks.rados.rados.0.smithi190.stdout:167: snap_remove snap 9 2023-11-01T04:16:34.196 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:34.196 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-01T04:16:34.220 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:34.221 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-01T04:16:34.255 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:34.255 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-01T04:16:34.282 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:34.279+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:34.286 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:34.286 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-01T04:16:34.308 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:34.308 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-01T04:16:34.317 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:34.317 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:16:34.317 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2023-11-01T04:16:34.318 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2023-11-01T04:16:34.318 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:34.331 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:34.331 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-01T04:16:34.333 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-01T04:16:34.351 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:34.351 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-01T04:16:34.368 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:34.368 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-01T04:16:34.383 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:34.378+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:34.387 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:34.388 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-01T04:16:34.407 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:34.408 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-01T04:16:34.441 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:34.442 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-01T04:16:34.462 INFO:teuthology.orchestra.run.smithi153.stdout:osd.12: {} 2023-11-01T04:16:34.462 INFO:teuthology.orchestra.run.smithi153.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-01T04:16:34.474 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:34.474 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:34.484 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:34.481+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:34.487 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-01T04:16:34.536 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:34.537 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-01T04:16:34.584 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:34.584+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:34.638 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:34.638 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:34.651 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-01T04:16:34.685 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:34.683+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:34.747 INFO:teuthology.orchestra.run.smithi153.stderr:nodeep-scrub is set 2023-11-01T04:16:34.748 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.15 to 1 (10000) 2023-11-01T04:16:34.756 INFO:tasks.rados.rados.0.smithi190.stdout:167: done (6 left) 2023-11-01T04:16:34.756 INFO:tasks.rados.rados.0.smithi190.stdout:168: delete oid 31 current snap is 13 2023-11-01T04:16:34.756 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 32 v 39 (ObjNum 53 snap 1 seq_num 53) dirty exists 2023-11-01T04:16:34.756 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 28 v 37 (ObjNum 27 snap 0 seq_num 27) dirty exists 2023-11-01T04:16:34.757 INFO:tasks.rados.rados.0.smithi190.stdout:162: finishing copy_from racing read to smithi19039307-33 2023-11-01T04:16:34.757 INFO:tasks.rados.rados.0.smithi190.stdout:159: expect (ObjNum 71 snap 8 seq_num 71) 2023-11-01T04:16:34.780 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-11-01T04:16:34.780 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-01T04:16:34.785 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:34.782+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:34.829 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:34.830 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:34.833 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:34.833 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-01T04:16:34.843 INFO:tasks.rados.rados.0.smithi190.stdout:162: finishing copy_from to smithi19039307-33 2023-11-01T04:16:34.843 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 33 v 4 (ObjNum 15 snap 0 seq_num 15) dirty exists 2023-11-01T04:16:34.844 INFO:tasks.rados.rados.0.smithi190.stdout:165: finishing write tid 1 to smithi19039307-25 2023-11-01T04:16:34.844 INFO:tasks.rados.rados.0.smithi190.stdout:166: finishing write tid 1 to smithi19039307-35 2023-11-01T04:16:34.844 INFO:tasks.rados.rados.0.smithi190.stdout:166: finishing write tid 2 to smithi19039307-35 2023-11-01T04:16:34.844 INFO:tasks.rados.rados.0.smithi190.stdout:165: finishing write tid 2 to smithi19039307-25 2023-11-01T04:16:34.845 INFO:tasks.rados.rados.0.smithi190.stdout:165: finishing write tid 3 to smithi19039307-25 2023-11-01T04:16:34.845 INFO:tasks.rados.rados.0.smithi190.stdout:165: finishing write tid 4 to smithi19039307-25 2023-11-01T04:16:34.846 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-01T04:16:34.853 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:34.853 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-01T04:16:34.916 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:34.916+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:34.957 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:34.958 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:34.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-01T04:16:35.016 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:35.016+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:35.033 INFO:tasks.rados.rados.0.smithi190.stdout:165: finishing write tid 5 to smithi19039307-25 2023-11-01T04:16:35.033 INFO:tasks.rados.rados.0.smithi190.stdout:165: oid 25 updating version 0 to 31 2023-11-01T04:16:35.033 INFO:tasks.rados.rados.0.smithi190.stdout:165: oid 25 version 31 is already newer than 29 2023-11-01T04:16:35.033 INFO:tasks.rados.rados.0.smithi190.stdout:165: oid 25 version 31 is already newer than 30 2023-11-01T04:16:35.034 INFO:tasks.rados.rados.0.smithi190.stdout:165: oid 25 updating version 31 to 32 2023-11-01T04:16:35.034 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 25 v 32 (ObjNum 76 snap 13 seq_num 76) dirty exists 2023-11-01T04:16:35.034 INFO:tasks.rados.rados.0.smithi190.stdout:165: left oid 25 (ObjNum 76 snap 13 seq_num 76) 2023-11-01T04:16:35.076 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:16:35.078 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-11-01T04:16:35.088 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:35.086+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:35.094 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2023-11-01T04:16:34.752181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":44417568,"num_objects":83,"num_object_clones":32,"num_object_copies":571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":2,"num_read":371,"num_read_kb":35859,"num_write":484,"num_write_kb":50532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":489,"ondisk_log_size":489,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":516240,"kb_used_data":83864,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509433200,"statfs":{"total":1546188226560,"available":1545659596800,"internally_reserved":0,"allocated":85876736,"data_stored":97331100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25505,"internal_metadata":442473567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10764288,"num_objects":7,"num_object_clones":7,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":-2,"num_read":21,"num_read_kb":3993,"num_write":22,"num_write_kb":9792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.687964"},"pg_stats":[{"pgid":"3.f","version":"45'10","reported_seq":79,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.337744+0000","last_change":"2023-11-01T04:16:24.958302+0000","last_active":"2023-11-01T04:16:32.337744+0000","last_peered":"2023-11-01T04:16:32.337744+0000","last_clean":"2023-11-01T04:16:32.337744+0000","last_became_active":"2023-11-01T04:16:11.171434+0000","last_became_peered":"2023-11-01T04:16:11.171434+0000","last_unstale":"2023-11-01T04:16:32.337744+0000","last_undegraded":"2023-11-01T04:16:32.337744+0000","last_fullsized":"2023-11-01T04:16:32.337744+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:10.036116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091376499999999998,"stat_sum":{"num_bytes":557056,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":1666,"num_write":24,"num_write_kb":544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,15,13,14,11,2],"acting":[0,3,15,13,14,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.e","version":"47'43","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.341902+0000","last_change":"2023-11-01T04:16:25.754268+0000","last_active":"2023-11-01T04:16:32.341902+0000","last_peered":"2023-11-01T04:16:32.341902+0000","last_clean":"2023-11-01T04:16:32.341902+0000","last_became_active":"2023-11-01T04:16:11.168562+0000","last_became_peered":"2023-11-01T04:16:11.168562+0000","last_unstale":"2023-11-01T04:16:32.341902+0000","last_undegraded":"2023-11-01T04:16:32.341902+0000","last_fullsized":"2023-11-01T04:16:32.341902+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:39.053537+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.81937162399999997,"stat_sum":{"num_bytes":3342336,"num_objects":8,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":22,"num_read_kb":4088,"num_write":22,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1,4,2,3,5],"acting":[11,9,1,4,2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.d","version":"50'35","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.183261+0000","last_change":"2023-11-01T04:16:24.976858+0000","last_active":"2023-11-01T04:16:32.183261+0000","last_peered":"2023-11-01T04:16:32.183261+0000","last_clean":"2023-11-01T04:16:32.183261+0000","last_became_active":"2023-11-01T04:16:11.267819+0000","last_became_peered":"2023-11-01T04:16:11.267819+0000","last_unstale":"2023-11-01T04:16:32.183261+0000","last_undegraded":"2023-11-01T04:16:32.183261+0000","last_fullsized":"2023-11-01T04:16:32.183261+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:11.512372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100917983,"stat_sum":{"num_bytes":7995392,"num_objects":11,"num_object_clones":4,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":28,"num_read_kb":2362,"num_write":43,"num_write_kb":8864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,11,0,8,2,6],"acting":[12,4,11,0,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.c","version":"33'2","reported_seq":73,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.337423+0000","last_change":"2023-11-01T04:16:24.858060+0000","last_active":"2023-11-01T04:16:32.337423+0000","last_peered":"2023-11-01T04:16:32.337423+0000","last_clean":"2023-11-01T04:16:32.337423+0000","last_became_active":"2023-11-01T04:16:11.169936+0000","last_became_peered":"2023-11-01T04:16:11.169936+0000","last_unstale":"2023-11-01T04:16:32.337423+0000","last_undegraded":"2023-11-01T04:16:32.337423+0000","last_fullsized":"2023-11-01T04:16:32.337423+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:36.163146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088480600000000003,"stat_sum":{"num_bytes":98304,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,12,3,7,2,6],"acting":[0,10,12,3,7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.b","version":"45'15","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.339019+0000","last_change":"2023-11-01T04:16:25.062131+0000","last_active":"2023-11-01T04:16:32.339019+0000","last_peered":"2023-11-01T04:16:32.339019+0000","last_clean":"2023-11-01T04:16:32.339019+0000","last_became_active":"2023-11-01T04:16:11.228189+0000","last_became_peered":"2023-11-01T04:16:11.228189+0000","last_unstale":"2023-11-01T04:16:32.339019+0000","last_undegraded":"2023-11-01T04:16:32.339019+0000","last_fullsized":"2023-11-01T04:16:32.339019+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:29.986722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.192567765,"stat_sum":{"num_bytes":983040,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":9,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1,9,11,4,14],"acting":[10,6,1,9,11,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.a","version":"47'17","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.183240+0000","last_change":"2023-11-01T04:16:24.776434+0000","last_active":"2023-11-01T04:16:32.183240+0000","last_peered":"2023-11-01T04:16:32.183240+0000","last_clean":"2023-11-01T04:16:32.183240+0000","last_became_active":"2023-11-01T04:16:11.103812+0000","last_became_peered":"2023-11-01T04:16:11.103812+0000","last_unstale":"2023-11-01T04:16:32.183240+0000","last_undegraded":"2023-11-01T04:16:32.183240+0000","last_fullsized":"2023-11-01T04:16:32.183240+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:36.307569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063471199999999995,"stat_sum":{"num_bytes":1687552,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":916,"num_write":9,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,1,9,3,15,8],"acting":[12,4,1,9,3,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.9","version":"51'37","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.441899+0000","last_change":"2023-11-01T04:16:24.852429+0000","last_active":"2023-11-01T04:16:32.441899+0000","last_peered":"2023-11-01T04:16:32.441899+0000","last_clean":"2023-11-01T04:16:32.441899+0000","last_became_active":"2023-11-01T04:16:10.995421+0000","last_became_peered":"2023-11-01T04:16:10.995421+0000","last_unstale":"2023-11-01T04:16:32.441899+0000","last_undegraded":"2023-11-01T04:16:32.441899+0000","last_fullsized":"2023-11-01T04:16:32.441899+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:13.618069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014725109999999999,"stat_sum":{"num_bytes":3932160,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":18,"num_read_kb":3176,"num_write":19,"num_write_kb":4768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,14,11,4,10,3],"acting":[7,1,14,11,4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.8","version":"46'22","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.438995+0000","last_change":"2023-11-01T04:16:24.887302+0000","last_active":"2023-11-01T04:16:32.438995+0000","last_peered":"2023-11-01T04:16:32.438995+0000","last_clean":"2023-11-01T04:16:32.438995+0000","last_became_active":"2023-11-01T04:16:11.114141+0000","last_became_peered":"2023-11-01T04:16:11.114141+0000","last_unstale":"2023-11-01T04:16:32.438995+0000","last_undegraded":"2023-11-01T04:16:32.438995+0000","last_fullsized":"2023-11-01T04:16:32.438995+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:29.310302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015188910000000001,"stat_sum":{"num_bytes":4046848,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":151,"num_write":15,"num_write_kb":3952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,11,3,14,0,7],"acting":[5,10,11,3,14,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"1.0","version":"33'157","reported_seq":234,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.106013+0000","last_change":"2023-11-01T04:15:59.016143+0000","last_active":"2023-11-01T04:16:31.106013+0000","last_peered":"2023-11-01T04:16:31.106013+0000","last_clean":"2023-11-01T04:16:31.106013+0000","last_became_active":"2023-11-01T04:15:59.014137+0000","last_became_peered":"2023-11-01T04:15:59.014137+0000","last_unstale":"2023-11-01T04:16:31.106013+0000","last_undegraded":"2023-11-01T04:16:31.106013+0000","last_fullsized":"2023-11-01T04:16:31.106013+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_clean_scrub_stamp":"2023-11-01T04:15:57.686019+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:15.188172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":35,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.238124+0000","last_change":"2023-11-01T04:16:02.077904+0000","last_active":"2023-11-01T04:16:31.238124+0000","last_peered":"2023-11-01T04:16:31.238124+0000","last_clean":"2023-11-01T04:16:31.238124+0000","last_became_active":"2023-11-01T04:16:02.077101+0000","last_became_peered":"2023-11-01T04:16:02.077101+0000","last_unstale":"2023-11-01T04:16:31.238124+0000","last_undegraded":"2023-11-01T04:16:31.238124+0000","last_fullsized":"2023-11-01T04:16:31.238124+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:10.452369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"50'40","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.187424+0000","last_change":"2023-11-01T04:16:25.040478+0000","last_active":"2023-11-01T04:16:32.187424+0000","last_peered":"2023-11-01T04:16:32.187424+0000","last_clean":"2023-11-01T04:16:32.187424+0000","last_became_active":"2023-11-01T04:16:11.100400+0000","last_became_peered":"2023-11-01T04:16:11.100400+0000","last_unstale":"2023-11-01T04:16:32.187424+0000","last_undegraded":"2023-11-01T04:16:32.187424+0000","last_fullsized":"2023-11-01T04:16:32.187424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:32.839951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.289448333,"stat_sum":{"num_bytes":5603328,"num_objects":8,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":9468,"num_write":26,"num_write_kb":5824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,3,0,4,8,1,7],"acting":[13,3,0,4,8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.187561+0000","last_change":"2023-11-01T04:16:01.930714+0000","last_active":"2023-11-01T04:16:31.187561+0000","last_peered":"2023-11-01T04:16:31.187561+0000","last_clean":"2023-11-01T04:16:31.187561+0000","last_became_active":"2023-11-01T04:16:01.930333+0000","last_became_peered":"2023-11-01T04:16:01.930333+0000","last_unstale":"2023-11-01T04:16:31.187561+0000","last_undegraded":"2023-11-01T04:16:31.187561+0000","last_fullsized":"2023-11-01T04:16:31.187561+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:06.734556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"47'24","reported_seq":107,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.331337+0000","last_change":"2023-11-01T04:16:25.270171+0000","last_active":"2023-11-01T04:16:31.331337+0000","last_peered":"2023-11-01T04:16:31.331337+0000","last_clean":"2023-11-01T04:16:31.331337+0000","last_became_active":"2023-11-01T04:16:11.197508+0000","last_became_peered":"2023-11-01T04:16:11.197508+0000","last_unstale":"2023-11-01T04:16:31.331337+0000","last_undegraded":"2023-11-01T04:16:31.331337+0000","last_fullsized":"2023-11-01T04:16:31.331337+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:39.284994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.49390870199999998,"stat_sum":{"num_bytes":3260416,"num_objects":9,"num_object_clones":4,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":28,"num_read_kb":2698,"num_write":36,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9,5,7,3,13],"acting":[6,4,9,5,7,3,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.184099+0000","last_change":"2023-11-01T04:16:01.757411+0000","last_active":"2023-11-01T04:16:31.184099+0000","last_peered":"2023-11-01T04:16:31.184099+0000","last_clean":"2023-11-01T04:16:31.184099+0000","last_became_active":"2023-11-01T04:16:01.756620+0000","last_became_peered":"2023-11-01T04:16:01.756620+0000","last_unstale":"2023-11-01T04:16:31.184099+0000","last_undegraded":"2023-11-01T04:16:31.184099+0000","last_fullsized":"2023-11-01T04:16:31.184099+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:02.829288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"43'12","reported_seq":91,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.441086+0000","last_change":"2023-11-01T04:16:24.760706+0000","last_active":"2023-11-01T04:16:32.441086+0000","last_peered":"2023-11-01T04:16:32.441086+0000","last_clean":"2023-11-01T04:16:32.441086+0000","last_became_active":"2023-11-01T04:16:11.014296+0000","last_became_peered":"2023-11-01T04:16:11.014296+0000","last_unstale":"2023-11-01T04:16:32.441086+0000","last_undegraded":"2023-11-01T04:16:32.441086+0000","last_fullsized":"2023-11-01T04:16:32.441086+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:27.608393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001427897,"stat_sum":{"num_bytes":917504,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":4949,"num_write":6,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4,1,5,15,6],"acting":[3,7,4,1,5,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":34,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.287795+0000","last_change":"2023-11-01T04:16:02.153588+0000","last_active":"2023-11-01T04:16:31.287795+0000","last_peered":"2023-11-01T04:16:31.287795+0000","last_clean":"2023-11-01T04:16:31.287795+0000","last_became_active":"2023-11-01T04:16:02.053124+0000","last_became_peered":"2023-11-01T04:16:02.053124+0000","last_unstale":"2023-11-01T04:16:31.287795+0000","last_undegraded":"2023-11-01T04:16:31.287795+0000","last_fullsized":"2023-11-01T04:16:31.287795+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:11.022108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"51'19","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.087398+0000","last_change":"2023-11-01T04:16:24.751930+0000","last_active":"2023-11-01T04:16:32.087398+0000","last_peered":"2023-11-01T04:16:32.087398+0000","last_clean":"2023-11-01T04:16:32.087398+0000","last_became_active":"2023-11-01T04:16:11.070168+0000","last_became_peered":"2023-11-01T04:16:11.070168+0000","last_unstale":"2023-11-01T04:16:32.087398+0000","last_undegraded":"2023-11-01T04:16:32.087398+0000","last_fullsized":"2023-11-01T04:16:32.087398+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:28.131127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001604381,"stat_sum":{"num_bytes":4046848,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":2744,"num_write":33,"num_write_kb":3952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8,2,0,11,5,12],"acting":[13,8,2,0,11,5,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.5","version":"34'2","reported_seq":70,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.340034+0000","last_change":"2023-11-01T04:16:24.958456+0000","last_active":"2023-11-01T04:16:32.340034+0000","last_peered":"2023-11-01T04:16:32.340034+0000","last_clean":"2023-11-01T04:16:32.340034+0000","last_became_active":"2023-11-01T04:16:11.168020+0000","last_became_peered":"2023-11-01T04:16:11.168020+0000","last_unstale":"2023-11-01T04:16:32.340034+0000","last_undegraded":"2023-11-01T04:16:32.340034+0000","last_fullsized":"2023-11-01T04:16:32.340034+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:30.030409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100865698,"stat_sum":{"num_bytes":344064,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,14,13,10,2,11],"acting":[8,6,14,13,10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-11-01T04:16:30.568469+0000","last_change":"2023-11-01T04:16:15.274405+0000","last_active":"2023-11-01T04:16:30.568469+0000","last_peered":"2023-11-01T04:16:30.568469+0000","last_clean":"2023-11-01T04:16:30.568469+0000","last_became_active":"2023-11-01T04:16:15.173499+0000","last_became_peered":"2023-11-01T04:16:15.173499+0000","last_unstale":"2023-11-01T04:16:30.568469+0000","last_undegraded":"2023-11-01T04:16:30.568469+0000","last_fullsized":"2023-11-01T04:16:30.568469+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:06.241660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.4","version":"48'23","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.439599+0000","last_change":"2023-11-01T04:16:24.873094+0000","last_active":"2023-11-01T04:16:32.439599+0000","last_peered":"2023-11-01T04:16:32.439599+0000","last_clean":"2023-11-01T04:16:32.439599+0000","last_became_active":"2023-11-01T04:16:11.207236+0000","last_became_peered":"2023-11-01T04:16:11.207236+0000","last_unstale":"2023-11-01T04:16:32.439599+0000","last_undegraded":"2023-11-01T04:16:32.439599+0000","last_fullsized":"2023-11-01T04:16:32.439599+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:27.464512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00291779,"stat_sum":{"num_bytes":2506752,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":2069,"num_write":14,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,15,14,1,6,3,13],"acting":[10,15,14,1,6,3,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":35,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.079209+0000","last_change":"2023-11-01T04:16:02.168355+0000","last_active":"2023-11-01T04:16:31.079209+0000","last_peered":"2023-11-01T04:16:31.079209+0000","last_clean":"2023-11-01T04:16:31.079209+0000","last_became_active":"2023-11-01T04:16:02.167473+0000","last_became_peered":"2023-11-01T04:16:02.167473+0000","last_unstale":"2023-11-01T04:16:31.079209+0000","last_undegraded":"2023-11-01T04:16:31.079209+0000","last_fullsized":"2023-11-01T04:16:31.079209+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:10.452369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"48'25","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.440886+0000","last_change":"2023-11-01T04:16:24.764246+0000","last_active":"2023-11-01T04:16:32.440886+0000","last_peered":"2023-11-01T04:16:32.440886+0000","last_clean":"2023-11-01T04:16:32.440886+0000","last_became_active":"2023-11-01T04:16:11.205734+0000","last_became_peered":"2023-11-01T04:16:11.205734+0000","last_unstale":"2023-11-01T04:16:32.440886+0000","last_undegraded":"2023-11-01T04:16:32.440886+0000","last_fullsized":"2023-11-01T04:16:32.440886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:38.691729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024236520000000001,"stat_sum":{"num_bytes":3145728,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":19,"num_read_kb":1447,"num_write":16,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,9,7,2,5,0],"acting":[14,11,9,7,2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.279356+0000","last_change":"2023-11-01T04:16:01.875964+0000","last_active":"2023-11-01T04:16:31.279356+0000","last_peered":"2023-11-01T04:16:31.279356+0000","last_clean":"2023-11-01T04:16:31.279356+0000","last_became_active":"2023-11-01T04:16:01.875255+0000","last_became_peered":"2023-11-01T04:16:01.875255+0000","last_unstale":"2023-11-01T04:16:31.279356+0000","last_undegraded":"2023-11-01T04:16:31.279356+0000","last_fullsized":"2023-11-01T04:16:31.279356+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:05.611210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11],"acting":[14,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.6","version":"47'6","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.339648+0000","last_change":"2023-11-01T04:16:24.762360+0000","last_active":"2023-11-01T04:16:32.339648+0000","last_peered":"2023-11-01T04:16:32.339648+0000","last_clean":"2023-11-01T04:16:32.339648+0000","last_became_active":"2023-11-01T04:16:11.103960+0000","last_became_peered":"2023-11-01T04:16:11.103960+0000","last_unstale":"2023-11-01T04:16:32.339648+0000","last_undegraded":"2023-11-01T04:16:32.339648+0000","last_fullsized":"2023-11-01T04:16:32.339648+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:36.412239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017753619999999999,"stat_sum":{"num_bytes":1359872,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":4,"num_read_kb":2,"num_write":5,"num_write_kb":1328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,1,8,12,0,9],"acting":[14,13,1,8,12,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":34,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.187997+0000","last_change":"2023-11-01T04:16:01.855207+0000","last_active":"2023-11-01T04:16:31.187997+0000","last_peered":"2023-11-01T04:16:31.187997+0000","last_clean":"2023-11-01T04:16:31.187997+0000","last_became_active":"2023-11-01T04:16:01.854811+0000","last_became_peered":"2023-11-01T04:16:01.854811+0000","last_unstale":"2023-11-01T04:16:31.187997+0000","last_undegraded":"2023-11-01T04:16:31.187997+0000","last_fullsized":"2023-11-01T04:16:31.187997+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:08.297413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":43827200,"num_objects":81,"num_object_clones":32,"num_object_copies":567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":2,"num_read":235,"num_read_kb":35741,"num_write":281,"num_write_kb":47072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":74031104,"data_stored":90603520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":332,"ondisk_log_size":332,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":22,"seq":94489280522,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29580,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342260,"statfs":{"total":96636764160,"available":96606474240,"internally_reserved":0,"allocated":2621440,"data_stored":2757974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,6,7,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1120000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]}]},{"osd":14,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31728,"kb_used_data":4708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340112,"statfs":{"total":96636764160,"available":96604274688,"internally_reserved":0,"allocated":4820992,"data_stored":5309238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,5,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2090000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]}]},{"osd":13,"up_from":22,"seq":94489280522,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32224,"kb_used_data":5204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339616,"statfs":{"total":96636764160,"available":96603766784,"internally_reserved":0,"allocated":5328896,"data_stored":6516052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.883}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.923}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7410000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.968}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3849999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4710000000000001}]}]},{"osd":6,"up_from":21,"seq":90194313227,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":4380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":4485120,"data_stored":5036856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[3,4,5,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1819999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":295.12700000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280522,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34056,"kb_used_data":7032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337784,"statfs":{"total":96636764160,"available":96601890816,"internally_reserved":0,"allocated":7200768,"data_stored":8397622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.272}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]}]},{"osd":1,"up_from":22,"seq":94489280522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":5372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":5500928,"data_stored":6349622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.335}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.395}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]}]},{"osd":2,"up_from":22,"seq":94489280523,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":5424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":5554176,"data_stored":5923638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.319}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]}]},{"osd":3,"up_from":22,"seq":94489280522,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33712,"kb_used_data":6688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338128,"statfs":{"total":96636764160,"available":96602243072,"internally_reserved":0,"allocated":6848512,"data_stored":8303414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4790000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.383}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.536}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.341}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]}]},{"osd":4,"up_from":22,"seq":94489280522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34096,"kb_used_data":7068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337744,"statfs":{"total":96636764160,"available":96601849856,"internally_reserved":0,"allocated":7237632,"data_stored":8428340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,5,6,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4079999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6000000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.472}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]}]},{"osd":5,"up_from":22,"seq":94489280522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":5112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":5234688,"data_stored":5788470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":154.78299999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.296}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.274}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6419999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.351}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]}]},{"osd":7,"up_from":22,"seq":94489280521,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":5480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":5611520,"data_stored":6644534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.585}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5129999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280523,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32636,"kb_used_data":5612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339204,"statfs":{"total":96636764160,"available":96603344896,"internally_reserved":0,"allocated":5746688,"data_stored":6624054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":197.03700000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]}]},{"osd":9,"up_from":22,"seq":94489280522,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30852,"kb_used_data":3828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340988,"statfs":{"total":96636764160,"available":96605171712,"internally_reserved":0,"allocated":3919872,"data_stored":4100918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[3,6,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97299999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2549999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3600000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4690000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5409999999999999}]}]},{"osd":10,"up_from":22,"seq":94489280522,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30520,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341320,"statfs":{"total":96636764160,"available":96605511680,"internally_reserved":0,"allocated":3579904,"data_stored":3904310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1919999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.357}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4279999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4750000000000001}]}]},{"osd":11,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34544,"kb_used_data":7520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337296,"statfs":{"total":96636764160,"available":96601391104,"internally_reserved":0,"allocated":7700480,"data_stored":8508214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9350000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7850000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.488}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2109999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.974}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4889999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1589999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4430000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1360000000000001}]}]},{"osd":12,"up_from":22,"seq":94489280523,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":4380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":4485120,"data_stored":4737844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]}],"pool_statfs":[{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6529024,"data_stored":8044544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4829184,"data_stored":5996544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4882432,"data_stored":5570560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6176768,"data_stored":7950336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6549504,"data_stored":8065024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4562944,"data_stored":5435392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3829760,"data_stored":4694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4939776,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":5074944,"data_stored":6270976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":3248128,"data_stored":3747840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2908160,"data_stored":3551232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":7028736,"data_stored":8155136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":3796992,"data_stored":4374528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4108288,"data_stored":5623808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4149248,"data_stored":4956160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":1417216,"data_stored":1875968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-01T04:16:35.098 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-01T04:16:35.102 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:35.102 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:35.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-01T04:16:35.125 INFO:tasks.rados.rados.0.smithi190.stdout:159: done (6 left) 2023-11-01T04:16:35.125 INFO:tasks.rados.rados.0.smithi190.stdout:161: done (5 left) 2023-11-01T04:16:35.126 INFO:tasks.rados.rados.0.smithi190.stdout:162: done (4 left) 2023-11-01T04:16:35.126 INFO:tasks.rados.rados.0.smithi190.stdout:163: done (3 left) 2023-11-01T04:16:35.126 INFO:tasks.rados.rados.0.smithi190.stdout:165: done (2 left) 2023-11-01T04:16:35.126 INFO:tasks.rados.rados.0.smithi190.stdout:168: done (1 left) 2023-11-01T04:16:35.127 INFO:tasks.rados.rados.0.smithi190.stdout:169: snap_create 2023-11-01T04:16:35.188 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:35.187+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:35.242 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:35.242 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:35.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-01T04:16:35.290 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:35.287+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:35.323 INFO:tasks.rados.rados.0.smithi190.stdout:166: finishing write tid 3 to smithi19039307-35 2023-11-01T04:16:35.324 INFO:tasks.rados.rados.0.smithi190.stdout:166: finishing write tid 4 to smithi19039307-35 2023-11-01T04:16:35.382 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:35.383 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:35.391 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:35.387+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:35.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-01T04:16:35.434 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:16:35.435 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-11-01T04:16:35.452 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2023-11-01T04:16:34.752181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":44417568,"num_objects":83,"num_object_clones":32,"num_object_copies":571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":2,"num_read":371,"num_read_kb":35859,"num_write":484,"num_write_kb":50532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":489,"ondisk_log_size":489,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":516240,"kb_used_data":83864,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509433200,"statfs":{"total":1546188226560,"available":1545659596800,"internally_reserved":0,"allocated":85876736,"data_stored":97331100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25505,"internal_metadata":442473567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10764288,"num_objects":7,"num_object_clones":7,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":-2,"num_read":21,"num_read_kb":3993,"num_write":22,"num_write_kb":9792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.687964"},"pg_stats":[{"pgid":"3.f","version":"45'10","reported_seq":79,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.337744+0000","last_change":"2023-11-01T04:16:24.958302+0000","last_active":"2023-11-01T04:16:32.337744+0000","last_peered":"2023-11-01T04:16:32.337744+0000","last_clean":"2023-11-01T04:16:32.337744+0000","last_became_active":"2023-11-01T04:16:11.171434+0000","last_became_peered":"2023-11-01T04:16:11.171434+0000","last_unstale":"2023-11-01T04:16:32.337744+0000","last_undegraded":"2023-11-01T04:16:32.337744+0000","last_fullsized":"2023-11-01T04:16:32.337744+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:10.036116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091376499999999998,"stat_sum":{"num_bytes":557056,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":1666,"num_write":24,"num_write_kb":544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,15,13,14,11,2],"acting":[0,3,15,13,14,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.e","version":"47'43","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.341902+0000","last_change":"2023-11-01T04:16:25.754268+0000","last_active":"2023-11-01T04:16:32.341902+0000","last_peered":"2023-11-01T04:16:32.341902+0000","last_clean":"2023-11-01T04:16:32.341902+0000","last_became_active":"2023-11-01T04:16:11.168562+0000","last_became_peered":"2023-11-01T04:16:11.168562+0000","last_unstale":"2023-11-01T04:16:32.341902+0000","last_undegraded":"2023-11-01T04:16:32.341902+0000","last_fullsized":"2023-11-01T04:16:32.341902+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:39.053537+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.81937162399999997,"stat_sum":{"num_bytes":3342336,"num_objects":8,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":22,"num_read_kb":4088,"num_write":22,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1,4,2,3,5],"acting":[11,9,1,4,2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.d","version":"50'35","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.183261+0000","last_change":"2023-11-01T04:16:24.976858+0000","last_active":"2023-11-01T04:16:32.183261+0000","last_peered":"2023-11-01T04:16:32.183261+0000","last_clean":"2023-11-01T04:16:32.183261+0000","last_became_active":"2023-11-01T04:16:11.267819+0000","last_became_peered":"2023-11-01T04:16:11.267819+0000","last_unstale":"2023-11-01T04:16:32.183261+0000","last_undegraded":"2023-11-01T04:16:32.183261+0000","last_fullsized":"2023-11-01T04:16:32.183261+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:11.512372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100917983,"stat_sum":{"num_bytes":7995392,"num_objects":11,"num_object_clones":4,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":28,"num_read_kb":2362,"num_write":43,"num_write_kb":8864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,11,0,8,2,6],"acting":[12,4,11,0,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.c","version":"33'2","reported_seq":73,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.337423+0000","last_change":"2023-11-01T04:16:24.858060+0000","last_active":"2023-11-01T04:16:32.337423+0000","last_peered":"2023-11-01T04:16:32.337423+0000","last_clean":"2023-11-01T04:16:32.337423+0000","last_became_active":"2023-11-01T04:16:11.169936+0000","last_became_peered":"2023-11-01T04:16:11.169936+0000","last_unstale":"2023-11-01T04:16:32.337423+0000","last_undegraded":"2023-11-01T04:16:32.337423+0000","last_fullsized":"2023-11-01T04:16:32.337423+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:36.163146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088480600000000003,"stat_sum":{"num_bytes":98304,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,12,3,7,2,6],"acting":[0,10,12,3,7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.b","version":"45'15","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.339019+0000","last_change":"2023-11-01T04:16:25.062131+0000","last_active":"2023-11-01T04:16:32.339019+0000","last_peered":"2023-11-01T04:16:32.339019+0000","last_clean":"2023-11-01T04:16:32.339019+0000","last_became_active":"2023-11-01T04:16:11.228189+0000","last_became_peered":"2023-11-01T04:16:11.228189+0000","last_unstale":"2023-11-01T04:16:32.339019+0000","last_undegraded":"2023-11-01T04:16:32.339019+0000","last_fullsized":"2023-11-01T04:16:32.339019+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:29.986722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.192567765,"stat_sum":{"num_bytes":983040,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":9,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1,9,11,4,14],"acting":[10,6,1,9,11,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.a","version":"47'17","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.183240+0000","last_change":"2023-11-01T04:16:24.776434+0000","last_active":"2023-11-01T04:16:32.183240+0000","last_peered":"2023-11-01T04:16:32.183240+0000","last_clean":"2023-11-01T04:16:32.183240+0000","last_became_active":"2023-11-01T04:16:11.103812+0000","last_became_peered":"2023-11-01T04:16:11.103812+0000","last_unstale":"2023-11-01T04:16:32.183240+0000","last_undegraded":"2023-11-01T04:16:32.183240+0000","last_fullsized":"2023-11-01T04:16:32.183240+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:36.307569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063471199999999995,"stat_sum":{"num_bytes":1687552,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":916,"num_write":9,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,1,9,3,15,8],"acting":[12,4,1,9,3,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.9","version":"51'37","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.441899+0000","last_change":"2023-11-01T04:16:24.852429+0000","last_active":"2023-11-01T04:16:32.441899+0000","last_peered":"2023-11-01T04:16:32.441899+0000","last_clean":"2023-11-01T04:16:32.441899+0000","last_became_active":"2023-11-01T04:16:10.995421+0000","last_became_peered":"2023-11-01T04:16:10.995421+0000","last_unstale":"2023-11-01T04:16:32.441899+0000","last_undegraded":"2023-11-01T04:16:32.441899+0000","last_fullsized":"2023-11-01T04:16:32.441899+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:13.618069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014725109999999999,"stat_sum":{"num_bytes":3932160,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":18,"num_read_kb":3176,"num_write":19,"num_write_kb":4768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,14,11,4,10,3],"acting":[7,1,14,11,4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.8","version":"46'22","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.438995+0000","last_change":"2023-11-01T04:16:24.887302+0000","last_active":"2023-11-01T04:16:32.438995+0000","last_peered":"2023-11-01T04:16:32.438995+0000","last_clean":"2023-11-01T04:16:32.438995+0000","last_became_active":"2023-11-01T04:16:11.114141+0000","last_became_peered":"2023-11-01T04:16:11.114141+0000","last_unstale":"2023-11-01T04:16:32.438995+0000","last_undegraded":"2023-11-01T04:16:32.438995+0000","last_fullsized":"2023-11-01T04:16:32.438995+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:29.310302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015188910000000001,"stat_sum":{"num_bytes":4046848,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":151,"num_write":15,"num_write_kb":3952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,11,3,14,0,7],"acting":[5,10,11,3,14,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"1.0","version":"33'157","reported_seq":234,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.106013+0000","last_change":"2023-11-01T04:15:59.016143+0000","last_active":"2023-11-01T04:16:31.106013+0000","last_peered":"2023-11-01T04:16:31.106013+0000","last_clean":"2023-11-01T04:16:31.106013+0000","last_became_active":"2023-11-01T04:15:59.014137+0000","last_became_peered":"2023-11-01T04:15:59.014137+0000","last_unstale":"2023-11-01T04:16:31.106013+0000","last_undegraded":"2023-11-01T04:16:31.106013+0000","last_fullsized":"2023-11-01T04:16:31.106013+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_clean_scrub_stamp":"2023-11-01T04:15:57.686019+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:15.188172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":35,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.238124+0000","last_change":"2023-11-01T04:16:02.077904+0000","last_active":"2023-11-01T04:16:31.238124+0000","last_peered":"2023-11-01T04:16:31.238124+0000","last_clean":"2023-11-01T04:16:31.238124+0000","last_became_active":"2023-11-01T04:16:02.077101+0000","last_became_peered":"2023-11-01T04:16:02.077101+0000","last_unstale":"2023-11-01T04:16:31.238124+0000","last_undegraded":"2023-11-01T04:16:31.238124+0000","last_fullsized":"2023-11-01T04:16:31.238124+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:10.452369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"50'40","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.187424+0000","last_change":"2023-11-01T04:16:25.040478+0000","last_active":"2023-11-01T04:16:32.187424+0000","last_peered":"2023-11-01T04:16:32.187424+0000","last_clean":"2023-11-01T04:16:32.187424+0000","last_became_active":"2023-11-01T04:16:11.100400+0000","last_became_peered":"2023-11-01T04:16:11.100400+0000","last_unstale":"2023-11-01T04:16:32.187424+0000","last_undegraded":"2023-11-01T04:16:32.187424+0000","last_fullsized":"2023-11-01T04:16:32.187424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:32.839951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.289448333,"stat_sum":{"num_bytes":5603328,"num_objects":8,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":9468,"num_write":26,"num_write_kb":5824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,3,0,4,8,1,7],"acting":[13,3,0,4,8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.187561+0000","last_change":"2023-11-01T04:16:01.930714+0000","last_active":"2023-11-01T04:16:31.187561+0000","last_peered":"2023-11-01T04:16:31.187561+0000","last_clean":"2023-11-01T04:16:31.187561+0000","last_became_active":"2023-11-01T04:16:01.930333+0000","last_became_peered":"2023-11-01T04:16:01.930333+0000","last_unstale":"2023-11-01T04:16:31.187561+0000","last_undegraded":"2023-11-01T04:16:31.187561+0000","last_fullsized":"2023-11-01T04:16:31.187561+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:06.734556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"47'24","reported_seq":107,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.331337+0000","last_change":"2023-11-01T04:16:25.270171+0000","last_active":"2023-11-01T04:16:31.331337+0000","last_peered":"2023-11-01T04:16:31.331337+0000","last_clean":"2023-11-01T04:16:31.331337+0000","last_became_active":"2023-11-01T04:16:11.197508+0000","last_became_peered":"2023-11-01T04:16:11.197508+0000","last_unstale":"2023-11-01T04:16:31.331337+0000","last_undegraded":"2023-11-01T04:16:31.331337+0000","last_fullsized":"2023-11-01T04:16:31.331337+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:39.284994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.49390870199999998,"stat_sum":{"num_bytes":3260416,"num_objects":9,"num_object_clones":4,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":28,"num_read_kb":2698,"num_write":36,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9,5,7,3,13],"acting":[6,4,9,5,7,3,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.184099+0000","last_change":"2023-11-01T04:16:01.757411+0000","last_active":"2023-11-01T04:16:31.184099+0000","last_peered":"2023-11-01T04:16:31.184099+0000","last_clean":"2023-11-01T04:16:31.184099+0000","last_became_active":"2023-11-01T04:16:01.756620+0000","last_became_peered":"2023-11-01T04:16:01.756620+0000","last_unstale":"2023-11-01T04:16:31.184099+0000","last_undegraded":"2023-11-01T04:16:31.184099+0000","last_fullsized":"2023-11-01T04:16:31.184099+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:02.829288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"43'12","reported_seq":91,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.441086+0000","last_change":"2023-11-01T04:16:24.760706+0000","last_active":"2023-11-01T04:16:32.441086+0000","last_peered":"2023-11-01T04:16:32.441086+0000","last_clean":"2023-11-01T04:16:32.441086+0000","last_became_active":"2023-11-01T04:16:11.014296+0000","last_became_peered":"2023-11-01T04:16:11.014296+0000","last_unstale":"2023-11-01T04:16:32.441086+0000","last_undegraded":"2023-11-01T04:16:32.441086+0000","last_fullsized":"2023-11-01T04:16:32.441086+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:27.608393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001427897,"stat_sum":{"num_bytes":917504,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":4949,"num_write":6,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4,1,5,15,6],"acting":[3,7,4,1,5,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":34,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.287795+0000","last_change":"2023-11-01T04:16:02.153588+0000","last_active":"2023-11-01T04:16:31.287795+0000","last_peered":"2023-11-01T04:16:31.287795+0000","last_clean":"2023-11-01T04:16:31.287795+0000","last_became_active":"2023-11-01T04:16:02.053124+0000","last_became_peered":"2023-11-01T04:16:02.053124+0000","last_unstale":"2023-11-01T04:16:31.287795+0000","last_undegraded":"2023-11-01T04:16:31.287795+0000","last_fullsized":"2023-11-01T04:16:31.287795+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:11.022108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"51'19","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.087398+0000","last_change":"2023-11-01T04:16:24.751930+0000","last_active":"2023-11-01T04:16:32.087398+0000","last_peered":"2023-11-01T04:16:32.087398+0000","last_clean":"2023-11-01T04:16:32.087398+0000","last_became_active":"2023-11-01T04:16:11.070168+0000","last_became_peered":"2023-11-01T04:16:11.070168+0000","last_unstale":"2023-11-01T04:16:32.087398+0000","last_undegraded":"2023-11-01T04:16:32.087398+0000","last_fullsized":"2023-11-01T04:16:32.087398+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:28.131127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001604381,"stat_sum":{"num_bytes":4046848,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":2744,"num_write":33,"num_write_kb":3952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8,2,0,11,5,12],"acting":[13,8,2,0,11,5,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.5","version":"34'2","reported_seq":70,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.340034+0000","last_change":"2023-11-01T04:16:24.958456+0000","last_active":"2023-11-01T04:16:32.340034+0000","last_peered":"2023-11-01T04:16:32.340034+0000","last_clean":"2023-11-01T04:16:32.340034+0000","last_became_active":"2023-11-01T04:16:11.168020+0000","last_became_peered":"2023-11-01T04:16:11.168020+0000","last_unstale":"2023-11-01T04:16:32.340034+0000","last_undegraded":"2023-11-01T04:16:32.340034+0000","last_fullsized":"2023-11-01T04:16:32.340034+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:30.030409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100865698,"stat_sum":{"num_bytes":344064,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,14,13,10,2,11],"acting":[8,6,14,13,10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-11-01T04:16:30.568469+0000","last_change":"2023-11-01T04:16:15.274405+0000","last_active":"2023-11-01T04:16:30.568469+0000","last_peered":"2023-11-01T04:16:30.568469+0000","last_clean":"2023-11-01T04:16:30.568469+0000","last_became_active":"2023-11-01T04:16:15.173499+0000","last_became_peered":"2023-11-01T04:16:15.173499+0000","last_unstale":"2023-11-01T04:16:30.568469+0000","last_undegraded":"2023-11-01T04:16:30.568469+0000","last_fullsized":"2023-11-01T04:16:30.568469+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:06.241660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.4","version":"48'23","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.439599+0000","last_change":"2023-11-01T04:16:24.873094+0000","last_active":"2023-11-01T04:16:32.439599+0000","last_peered":"2023-11-01T04:16:32.439599+0000","last_clean":"2023-11-01T04:16:32.439599+0000","last_became_active":"2023-11-01T04:16:11.207236+0000","last_became_peered":"2023-11-01T04:16:11.207236+0000","last_unstale":"2023-11-01T04:16:32.439599+0000","last_undegraded":"2023-11-01T04:16:32.439599+0000","last_fullsized":"2023-11-01T04:16:32.439599+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:27.464512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00291779,"stat_sum":{"num_bytes":2506752,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":2069,"num_write":14,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,15,14,1,6,3,13],"acting":[10,15,14,1,6,3,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":35,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.079209+0000","last_change":"2023-11-01T04:16:02.168355+0000","last_active":"2023-11-01T04:16:31.079209+0000","last_peered":"2023-11-01T04:16:31.079209+0000","last_clean":"2023-11-01T04:16:31.079209+0000","last_became_active":"2023-11-01T04:16:02.167473+0000","last_became_peered":"2023-11-01T04:16:02.167473+0000","last_unstale":"2023-11-01T04:16:31.079209+0000","last_undegraded":"2023-11-01T04:16:31.079209+0000","last_fullsized":"2023-11-01T04:16:31.079209+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:10.452369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"48'25","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.440886+0000","last_change":"2023-11-01T04:16:24.764246+0000","last_active":"2023-11-01T04:16:32.440886+0000","last_peered":"2023-11-01T04:16:32.440886+0000","last_clean":"2023-11-01T04:16:32.440886+0000","last_became_active":"2023-11-01T04:16:11.205734+0000","last_became_peered":"2023-11-01T04:16:11.205734+0000","last_unstale":"2023-11-01T04:16:32.440886+0000","last_undegraded":"2023-11-01T04:16:32.440886+0000","last_fullsized":"2023-11-01T04:16:32.440886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:38.691729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024236520000000001,"stat_sum":{"num_bytes":3145728,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":19,"num_read_kb":1447,"num_write":16,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,9,7,2,5,0],"acting":[14,11,9,7,2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.279356+0000","last_change":"2023-11-01T04:16:01.875964+0000","last_active":"2023-11-01T04:16:31.279356+0000","last_peered":"2023-11-01T04:16:31.279356+0000","last_clean":"2023-11-01T04:16:31.279356+0000","last_became_active":"2023-11-01T04:16:01.875255+0000","last_became_peered":"2023-11-01T04:16:01.875255+0000","last_unstale":"2023-11-01T04:16:31.279356+0000","last_undegraded":"2023-11-01T04:16:31.279356+0000","last_fullsized":"2023-11-01T04:16:31.279356+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:05.611210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11],"acting":[14,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.6","version":"47'6","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.339648+0000","last_change":"2023-11-01T04:16:24.762360+0000","last_active":"2023-11-01T04:16:32.339648+0000","last_peered":"2023-11-01T04:16:32.339648+0000","last_clean":"2023-11-01T04:16:32.339648+0000","last_became_active":"2023-11-01T04:16:11.103960+0000","last_became_peered":"2023-11-01T04:16:11.103960+0000","last_unstale":"2023-11-01T04:16:32.339648+0000","last_undegraded":"2023-11-01T04:16:32.339648+0000","last_fullsized":"2023-11-01T04:16:32.339648+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:36.412239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017753619999999999,"stat_sum":{"num_bytes":1359872,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":4,"num_read_kb":2,"num_write":5,"num_write_kb":1328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,1,8,12,0,9],"acting":[14,13,1,8,12,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":34,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.187997+0000","last_change":"2023-11-01T04:16:01.855207+0000","last_active":"2023-11-01T04:16:31.187997+0000","last_peered":"2023-11-01T04:16:31.187997+0000","last_clean":"2023-11-01T04:16:31.187997+0000","last_became_active":"2023-11-01T04:16:01.854811+0000","last_became_peered":"2023-11-01T04:16:01.854811+0000","last_unstale":"2023-11-01T04:16:31.187997+0000","last_undegraded":"2023-11-01T04:16:31.187997+0000","last_fullsized":"2023-11-01T04:16:31.187997+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:08.297413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":43827200,"num_objects":81,"num_object_clones":32,"num_object_copies":567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":2,"num_read":235,"num_read_kb":35741,"num_write":281,"num_write_kb":47072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":74031104,"data_stored":90603520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":332,"ondisk_log_size":332,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":22,"seq":94489280522,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29580,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342260,"statfs":{"total":96636764160,"available":96606474240,"internally_reserved":0,"allocated":2621440,"data_stored":2757974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,6,7,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1120000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]}]},{"osd":14,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31728,"kb_used_data":4708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340112,"statfs":{"total":96636764160,"available":96604274688,"internally_reserved":0,"allocated":4820992,"data_stored":5309238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,5,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2090000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]}]},{"osd":13,"up_from":22,"seq":94489280522,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32224,"kb_used_data":5204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339616,"statfs":{"total":96636764160,"available":96603766784,"internally_reserved":0,"allocated":5328896,"data_stored":6516052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.883}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.923}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7410000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.968}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3849999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4710000000000001}]}]},{"osd":6,"up_from":21,"seq":90194313227,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":4380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":4485120,"data_stored":5036856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[3,4,5,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1819999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":295.12700000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280522,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34056,"kb_used_data":7032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337784,"statfs":{"total":96636764160,"available":96601890816,"internally_reserved":0,"allocated":7200768,"data_stored":8397622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.272}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]}]},{"osd":1,"up_from":22,"seq":94489280522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":5372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":5500928,"data_stored":6349622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.335}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.395}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]}]},{"osd":2,"up_from":22,"seq":94489280523,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":5424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":5554176,"data_stored":5923638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.319}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]}]},{"osd":3,"up_from":22,"seq":94489280522,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33712,"kb_used_data":6688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338128,"statfs":{"total":96636764160,"available":96602243072,"internally_reserved":0,"allocated":6848512,"data_stored":8303414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4790000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.383}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.536}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.341}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]}]},{"osd":4,"up_from":22,"seq":94489280522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34096,"kb_used_data":7068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337744,"statfs":{"total":96636764160,"available":96601849856,"internally_reserved":0,"allocated":7237632,"data_stored":8428340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,5,6,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4079999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6000000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.472}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]}]},{"osd":5,"up_from":22,"seq":94489280522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":5112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":5234688,"data_stored":5788470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":154.78299999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.296}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.274}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6419999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.351}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]}]},{"osd":7,"up_from":22,"seq":94489280521,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":5480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":5611520,"data_stored":6644534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.585}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5129999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280523,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32636,"kb_used_data":5612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339204,"statfs":{"total":96636764160,"available":96603344896,"internally_reserved":0,"allocated":5746688,"data_stored":6624054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":197.03700000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]}]},{"osd":9,"up_from":22,"seq":94489280522,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30852,"kb_used_data":3828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340988,"statfs":{"total":96636764160,"available":96605171712,"internally_reserved":0,"allocated":3919872,"data_stored":4100918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[3,6,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97299999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2549999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3600000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4690000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5409999999999999}]}]},{"osd":10,"up_from":22,"seq":94489280522,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30520,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341320,"statfs":{"total":96636764160,"available":96605511680,"internally_reserved":0,"allocated":3579904,"data_stored":3904310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1919999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.357}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4279999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4750000000000001}]}]},{"osd":11,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34544,"kb_used_data":7520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337296,"statfs":{"total":96636764160,"available":96601391104,"internally_reserved":0,"allocated":7700480,"data_stored":8508214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9350000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7850000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.488}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2109999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.974}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4889999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1589999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4430000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1360000000000001}]}]},{"osd":12,"up_from":22,"seq":94489280523,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":4380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":4485120,"data_stored":4737844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]}],"pool_statfs":[{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6529024,"data_stored":8044544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4829184,"data_stored":5996544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4882432,"data_stored":5570560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6176768,"data_stored":7950336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6549504,"data_stored":8065024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4562944,"data_stored":5435392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3829760,"data_stored":4694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4939776,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":5074944,"data_stored":6270976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":3248128,"data_stored":3747840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2908160,"data_stored":3551232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":7028736,"data_stored":8155136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":3796992,"data_stored":4374528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4108288,"data_stored":5623808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4149248,"data_stored":4956160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":1417216,"data_stored":1875968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-01T04:16:35.455 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-11-01T04:16:35.456 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-01T04:16:35.456 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2023-11-01T04:16:35.457 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-01T04:16:35.491 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:35.486+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:35.518 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:35.518 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:35.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-01T04:16:35.592 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:35.592+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:35.630 INFO:tasks.rados.rados.0.smithi190.stdout:166: finishing write tid 5 to smithi19039307-35 2023-11-01T04:16:35.630 INFO:tasks.rados.rados.0.smithi190.stdout:166: oid 35 updating version 0 to 43 2023-11-01T04:16:35.631 INFO:tasks.rados.rados.0.smithi190.stdout:166: oid 35 updating version 43 to 44 2023-11-01T04:16:35.631 INFO:tasks.rados.rados.0.smithi190.stdout:166: oid 35 updating version 44 to 45 2023-11-01T04:16:35.631 INFO:tasks.rados.rados.0.smithi190.stdout:166: oid 35 updating version 45 to 46 2023-11-01T04:16:35.631 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 35 v 46 (ObjNum 77 snap 13 seq_num 77) dirty exists 2023-11-01T04:16:35.632 INFO:tasks.rados.rados.0.smithi190.stdout:166: left oid 35 (ObjNum 77 snap 13 seq_num 77) 2023-11-01T04:16:35.657 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:35.658 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:35.672 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-01T04:16:35.693 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:35.690+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:35.741 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:16:35.742 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-11-01T04:16:35.758 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2023-11-01T04:16:35.614177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":44417568,"num_objects":83,"num_object_clones":32,"num_object_copies":571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":2,"num_read":371,"num_read_kb":35859,"num_write":484,"num_write_kb":50532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":489,"ondisk_log_size":489,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":516240,"kb_used_data":83864,"kb_used_omap":24,"kb_used_meta":432103,"kb_avail":1509433200,"statfs":{"total":1546188226560,"available":1545659596800,"internally_reserved":0,"allocated":85876736,"data_stored":97331100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25505,"internal_metadata":442473567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10764288,"num_objects":7,"num_object_clones":7,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":-2,"num_read":21,"num_read_kb":3993,"num_write":22,"num_write_kb":9792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001211"},"pg_stats":[{"pgid":"3.f","version":"45'10","reported_seq":79,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.337744+0000","last_change":"2023-11-01T04:16:24.958302+0000","last_active":"2023-11-01T04:16:32.337744+0000","last_peered":"2023-11-01T04:16:32.337744+0000","last_clean":"2023-11-01T04:16:32.337744+0000","last_became_active":"2023-11-01T04:16:11.171434+0000","last_became_peered":"2023-11-01T04:16:11.171434+0000","last_unstale":"2023-11-01T04:16:32.337744+0000","last_undegraded":"2023-11-01T04:16:32.337744+0000","last_fullsized":"2023-11-01T04:16:32.337744+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:10.036116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091376499999999998,"stat_sum":{"num_bytes":557056,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":1666,"num_write":24,"num_write_kb":544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,15,13,14,11,2],"acting":[0,3,15,13,14,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.e","version":"47'43","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.341902+0000","last_change":"2023-11-01T04:16:25.754268+0000","last_active":"2023-11-01T04:16:32.341902+0000","last_peered":"2023-11-01T04:16:32.341902+0000","last_clean":"2023-11-01T04:16:32.341902+0000","last_became_active":"2023-11-01T04:16:11.168562+0000","last_became_peered":"2023-11-01T04:16:11.168562+0000","last_unstale":"2023-11-01T04:16:32.341902+0000","last_undegraded":"2023-11-01T04:16:32.341902+0000","last_fullsized":"2023-11-01T04:16:32.341902+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:39.053537+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.81937162399999997,"stat_sum":{"num_bytes":3342336,"num_objects":8,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":22,"num_read_kb":4088,"num_write":22,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1,4,2,3,5],"acting":[11,9,1,4,2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.d","version":"50'35","reported_seq":120,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.183261+0000","last_change":"2023-11-01T04:16:24.976858+0000","last_active":"2023-11-01T04:16:32.183261+0000","last_peered":"2023-11-01T04:16:32.183261+0000","last_clean":"2023-11-01T04:16:32.183261+0000","last_became_active":"2023-11-01T04:16:11.267819+0000","last_became_peered":"2023-11-01T04:16:11.267819+0000","last_unstale":"2023-11-01T04:16:32.183261+0000","last_undegraded":"2023-11-01T04:16:32.183261+0000","last_fullsized":"2023-11-01T04:16:32.183261+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:11.512372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100917983,"stat_sum":{"num_bytes":7995392,"num_objects":11,"num_object_clones":4,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":28,"num_read_kb":2362,"num_write":43,"num_write_kb":8864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,11,0,8,2,6],"acting":[12,4,11,0,8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.c","version":"33'2","reported_seq":73,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.337423+0000","last_change":"2023-11-01T04:16:24.858060+0000","last_active":"2023-11-01T04:16:32.337423+0000","last_peered":"2023-11-01T04:16:32.337423+0000","last_clean":"2023-11-01T04:16:32.337423+0000","last_became_active":"2023-11-01T04:16:11.169936+0000","last_became_peered":"2023-11-01T04:16:11.169936+0000","last_unstale":"2023-11-01T04:16:32.337423+0000","last_undegraded":"2023-11-01T04:16:32.337423+0000","last_fullsized":"2023-11-01T04:16:32.337423+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:36.163146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088480600000000003,"stat_sum":{"num_bytes":98304,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,12,3,7,2,6],"acting":[0,10,12,3,7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.b","version":"45'15","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.339019+0000","last_change":"2023-11-01T04:16:25.062131+0000","last_active":"2023-11-01T04:16:32.339019+0000","last_peered":"2023-11-01T04:16:32.339019+0000","last_clean":"2023-11-01T04:16:32.339019+0000","last_became_active":"2023-11-01T04:16:11.228189+0000","last_became_peered":"2023-11-01T04:16:11.228189+0000","last_unstale":"2023-11-01T04:16:32.339019+0000","last_undegraded":"2023-11-01T04:16:32.339019+0000","last_fullsized":"2023-11-01T04:16:32.339019+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:29.986722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.192567765,"stat_sum":{"num_bytes":983040,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":9,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1,9,11,4,14],"acting":[10,6,1,9,11,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.a","version":"47'17","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.183240+0000","last_change":"2023-11-01T04:16:24.776434+0000","last_active":"2023-11-01T04:16:32.183240+0000","last_peered":"2023-11-01T04:16:32.183240+0000","last_clean":"2023-11-01T04:16:32.183240+0000","last_became_active":"2023-11-01T04:16:11.103812+0000","last_became_peered":"2023-11-01T04:16:11.103812+0000","last_unstale":"2023-11-01T04:16:32.183240+0000","last_undegraded":"2023-11-01T04:16:32.183240+0000","last_fullsized":"2023-11-01T04:16:32.183240+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:36.307569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063471199999999995,"stat_sum":{"num_bytes":1687552,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":916,"num_write":9,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,1,9,3,15,8],"acting":[12,4,1,9,3,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.9","version":"51'37","reported_seq":112,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.441899+0000","last_change":"2023-11-01T04:16:24.852429+0000","last_active":"2023-11-01T04:16:32.441899+0000","last_peered":"2023-11-01T04:16:32.441899+0000","last_clean":"2023-11-01T04:16:32.441899+0000","last_became_active":"2023-11-01T04:16:10.995421+0000","last_became_peered":"2023-11-01T04:16:10.995421+0000","last_unstale":"2023-11-01T04:16:32.441899+0000","last_undegraded":"2023-11-01T04:16:32.441899+0000","last_fullsized":"2023-11-01T04:16:32.441899+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:13.618069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014725109999999999,"stat_sum":{"num_bytes":3932160,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":18,"num_read_kb":3176,"num_write":19,"num_write_kb":4768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,14,11,4,10,3],"acting":[7,1,14,11,4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.8","version":"46'22","reported_seq":99,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.438995+0000","last_change":"2023-11-01T04:16:24.887302+0000","last_active":"2023-11-01T04:16:32.438995+0000","last_peered":"2023-11-01T04:16:32.438995+0000","last_clean":"2023-11-01T04:16:32.438995+0000","last_became_active":"2023-11-01T04:16:11.114141+0000","last_became_peered":"2023-11-01T04:16:11.114141+0000","last_unstale":"2023-11-01T04:16:32.438995+0000","last_undegraded":"2023-11-01T04:16:32.438995+0000","last_fullsized":"2023-11-01T04:16:32.438995+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:29.310302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015188910000000001,"stat_sum":{"num_bytes":4046848,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":151,"num_write":15,"num_write_kb":3952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,11,3,14,0,7],"acting":[5,10,11,3,14,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"1.0","version":"33'157","reported_seq":234,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.106013+0000","last_change":"2023-11-01T04:15:59.016143+0000","last_active":"2023-11-01T04:16:31.106013+0000","last_peered":"2023-11-01T04:16:31.106013+0000","last_clean":"2023-11-01T04:16:31.106013+0000","last_became_active":"2023-11-01T04:15:59.014137+0000","last_became_peered":"2023-11-01T04:15:59.014137+0000","last_unstale":"2023-11-01T04:16:31.106013+0000","last_undegraded":"2023-11-01T04:16:31.106013+0000","last_fullsized":"2023-11-01T04:16:31.106013+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_clean_scrub_stamp":"2023-11-01T04:15:57.686019+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:15.188172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":35,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.238124+0000","last_change":"2023-11-01T04:16:02.077904+0000","last_active":"2023-11-01T04:16:31.238124+0000","last_peered":"2023-11-01T04:16:31.238124+0000","last_clean":"2023-11-01T04:16:31.238124+0000","last_became_active":"2023-11-01T04:16:02.077101+0000","last_became_peered":"2023-11-01T04:16:02.077101+0000","last_unstale":"2023-11-01T04:16:31.238124+0000","last_undegraded":"2023-11-01T04:16:31.238124+0000","last_fullsized":"2023-11-01T04:16:31.238124+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:10.452369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"50'40","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.187424+0000","last_change":"2023-11-01T04:16:25.040478+0000","last_active":"2023-11-01T04:16:32.187424+0000","last_peered":"2023-11-01T04:16:32.187424+0000","last_clean":"2023-11-01T04:16:32.187424+0000","last_became_active":"2023-11-01T04:16:11.100400+0000","last_became_peered":"2023-11-01T04:16:11.100400+0000","last_unstale":"2023-11-01T04:16:32.187424+0000","last_undegraded":"2023-11-01T04:16:32.187424+0000","last_fullsized":"2023-11-01T04:16:32.187424+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:32.839951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.289448333,"stat_sum":{"num_bytes":5603328,"num_objects":8,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":9468,"num_write":26,"num_write_kb":5824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,3,0,4,8,1,7],"acting":[13,3,0,4,8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.187561+0000","last_change":"2023-11-01T04:16:01.930714+0000","last_active":"2023-11-01T04:16:31.187561+0000","last_peered":"2023-11-01T04:16:31.187561+0000","last_clean":"2023-11-01T04:16:31.187561+0000","last_became_active":"2023-11-01T04:16:01.930333+0000","last_became_peered":"2023-11-01T04:16:01.930333+0000","last_unstale":"2023-11-01T04:16:31.187561+0000","last_undegraded":"2023-11-01T04:16:31.187561+0000","last_fullsized":"2023-11-01T04:16:31.187561+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:06.734556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"47'24","reported_seq":107,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.331337+0000","last_change":"2023-11-01T04:16:25.270171+0000","last_active":"2023-11-01T04:16:31.331337+0000","last_peered":"2023-11-01T04:16:31.331337+0000","last_clean":"2023-11-01T04:16:31.331337+0000","last_became_active":"2023-11-01T04:16:11.197508+0000","last_became_peered":"2023-11-01T04:16:11.197508+0000","last_unstale":"2023-11-01T04:16:31.331337+0000","last_undegraded":"2023-11-01T04:16:31.331337+0000","last_fullsized":"2023-11-01T04:16:31.331337+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:39.284994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.49390870199999998,"stat_sum":{"num_bytes":3260416,"num_objects":9,"num_object_clones":4,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":28,"num_read_kb":2698,"num_write":36,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9,5,7,3,13],"acting":[6,4,9,5,7,3,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.184099+0000","last_change":"2023-11-01T04:16:01.757411+0000","last_active":"2023-11-01T04:16:31.184099+0000","last_peered":"2023-11-01T04:16:31.184099+0000","last_clean":"2023-11-01T04:16:31.184099+0000","last_became_active":"2023-11-01T04:16:01.756620+0000","last_became_peered":"2023-11-01T04:16:01.756620+0000","last_unstale":"2023-11-01T04:16:31.184099+0000","last_undegraded":"2023-11-01T04:16:31.184099+0000","last_fullsized":"2023-11-01T04:16:31.184099+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:02.829288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"43'12","reported_seq":91,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.441086+0000","last_change":"2023-11-01T04:16:24.760706+0000","last_active":"2023-11-01T04:16:32.441086+0000","last_peered":"2023-11-01T04:16:32.441086+0000","last_clean":"2023-11-01T04:16:32.441086+0000","last_became_active":"2023-11-01T04:16:11.014296+0000","last_became_peered":"2023-11-01T04:16:11.014296+0000","last_unstale":"2023-11-01T04:16:32.441086+0000","last_undegraded":"2023-11-01T04:16:32.441086+0000","last_fullsized":"2023-11-01T04:16:32.441086+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:27.608393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001427897,"stat_sum":{"num_bytes":917504,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":4949,"num_write":6,"num_write_kb":896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4,1,5,15,6],"acting":[3,7,4,1,5,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":34,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.287795+0000","last_change":"2023-11-01T04:16:02.153588+0000","last_active":"2023-11-01T04:16:31.287795+0000","last_peered":"2023-11-01T04:16:31.287795+0000","last_clean":"2023-11-01T04:16:31.287795+0000","last_became_active":"2023-11-01T04:16:02.053124+0000","last_became_peered":"2023-11-01T04:16:02.053124+0000","last_unstale":"2023-11-01T04:16:31.287795+0000","last_undegraded":"2023-11-01T04:16:31.287795+0000","last_fullsized":"2023-11-01T04:16:31.287795+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:11.022108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"51'19","reported_seq":100,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.087398+0000","last_change":"2023-11-01T04:16:24.751930+0000","last_active":"2023-11-01T04:16:32.087398+0000","last_peered":"2023-11-01T04:16:32.087398+0000","last_clean":"2023-11-01T04:16:32.087398+0000","last_became_active":"2023-11-01T04:16:11.070168+0000","last_became_peered":"2023-11-01T04:16:11.070168+0000","last_unstale":"2023-11-01T04:16:32.087398+0000","last_undegraded":"2023-11-01T04:16:32.087398+0000","last_fullsized":"2023-11-01T04:16:32.087398+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:28.131127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001604381,"stat_sum":{"num_bytes":4046848,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":2744,"num_write":33,"num_write_kb":3952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8,2,0,11,5,12],"acting":[13,8,2,0,11,5,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"3.5","version":"34'2","reported_seq":70,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.340034+0000","last_change":"2023-11-01T04:16:24.958456+0000","last_active":"2023-11-01T04:16:32.340034+0000","last_peered":"2023-11-01T04:16:32.340034+0000","last_clean":"2023-11-01T04:16:32.340034+0000","last_became_active":"2023-11-01T04:16:11.168020+0000","last_became_peered":"2023-11-01T04:16:11.168020+0000","last_unstale":"2023-11-01T04:16:32.340034+0000","last_undegraded":"2023-11-01T04:16:32.340034+0000","last_fullsized":"2023-11-01T04:16:32.340034+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:30.030409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100865698,"stat_sum":{"num_bytes":344064,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,14,13,10,2,11],"acting":[8,6,14,13,10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-11-01T04:16:30.568469+0000","last_change":"2023-11-01T04:16:15.274405+0000","last_active":"2023-11-01T04:16:30.568469+0000","last_peered":"2023-11-01T04:16:30.568469+0000","last_clean":"2023-11-01T04:16:30.568469+0000","last_became_active":"2023-11-01T04:16:15.173499+0000","last_became_peered":"2023-11-01T04:16:15.173499+0000","last_unstale":"2023-11-01T04:16:30.568469+0000","last_undegraded":"2023-11-01T04:16:30.568469+0000","last_fullsized":"2023-11-01T04:16:30.568469+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:06.241660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.4","version":"48'23","reported_seq":98,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.439599+0000","last_change":"2023-11-01T04:16:24.873094+0000","last_active":"2023-11-01T04:16:32.439599+0000","last_peered":"2023-11-01T04:16:32.439599+0000","last_clean":"2023-11-01T04:16:32.439599+0000","last_became_active":"2023-11-01T04:16:11.207236+0000","last_became_peered":"2023-11-01T04:16:11.207236+0000","last_unstale":"2023-11-01T04:16:32.439599+0000","last_undegraded":"2023-11-01T04:16:32.439599+0000","last_fullsized":"2023-11-01T04:16:32.439599+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:27.464512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00291779,"stat_sum":{"num_bytes":2506752,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":2069,"num_write":14,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,15,14,1,6,3,13],"acting":[10,15,14,1,6,3,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":35,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.079209+0000","last_change":"2023-11-01T04:16:02.168355+0000","last_active":"2023-11-01T04:16:31.079209+0000","last_peered":"2023-11-01T04:16:31.079209+0000","last_clean":"2023-11-01T04:16:31.079209+0000","last_became_active":"2023-11-01T04:16:02.167473+0000","last_became_peered":"2023-11-01T04:16:02.167473+0000","last_unstale":"2023-11-01T04:16:31.079209+0000","last_undegraded":"2023-11-01T04:16:31.079209+0000","last_fullsized":"2023-11-01T04:16:31.079209+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:10.452369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"48'25","reported_seq":103,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.440886+0000","last_change":"2023-11-01T04:16:24.764246+0000","last_active":"2023-11-01T04:16:32.440886+0000","last_peered":"2023-11-01T04:16:32.440886+0000","last_clean":"2023-11-01T04:16:32.440886+0000","last_became_active":"2023-11-01T04:16:11.205734+0000","last_became_peered":"2023-11-01T04:16:11.205734+0000","last_unstale":"2023-11-01T04:16:32.440886+0000","last_undegraded":"2023-11-01T04:16:32.440886+0000","last_fullsized":"2023-11-01T04:16:32.440886+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:38.691729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024236520000000001,"stat_sum":{"num_bytes":3145728,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":19,"num_read_kb":1447,"num_write":16,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,9,7,2,5,0],"acting":[14,11,9,7,2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.279356+0000","last_change":"2023-11-01T04:16:01.875964+0000","last_active":"2023-11-01T04:16:31.279356+0000","last_peered":"2023-11-01T04:16:31.279356+0000","last_clean":"2023-11-01T04:16:31.279356+0000","last_became_active":"2023-11-01T04:16:01.875255+0000","last_became_peered":"2023-11-01T04:16:01.875255+0000","last_unstale":"2023-11-01T04:16:31.279356+0000","last_undegraded":"2023-11-01T04:16:31.279356+0000","last_fullsized":"2023-11-01T04:16:31.279356+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:05.611210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11],"acting":[14,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.6","version":"47'6","reported_seq":77,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-11-01T04:16:32.339648+0000","last_change":"2023-11-01T04:16:24.762360+0000","last_active":"2023-11-01T04:16:32.339648+0000","last_peered":"2023-11-01T04:16:32.339648+0000","last_clean":"2023-11-01T04:16:32.339648+0000","last_became_active":"2023-11-01T04:16:11.103960+0000","last_became_peered":"2023-11-01T04:16:11.103960+0000","last_unstale":"2023-11-01T04:16:32.339648+0000","last_undegraded":"2023-11-01T04:16:32.339648+0000","last_fullsized":"2023-11-01T04:16:32.339648+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:36.412239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017753619999999999,"stat_sum":{"num_bytes":1359872,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":4,"num_read_kb":2,"num_write":5,"num_write_kb":1328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,1,8,12,0,9],"acting":[14,13,1,8,12,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":34,"reported_epoch":51,"state":"active+clean","last_fresh":"2023-11-01T04:16:31.187997+0000","last_change":"2023-11-01T04:16:01.855207+0000","last_active":"2023-11-01T04:16:31.187997+0000","last_peered":"2023-11-01T04:16:31.187997+0000","last_clean":"2023-11-01T04:16:31.187997+0000","last_became_active":"2023-11-01T04:16:01.854811+0000","last_became_peered":"2023-11-01T04:16:01.854811+0000","last_unstale":"2023-11-01T04:16:31.187997+0000","last_undegraded":"2023-11-01T04:16:31.187997+0000","last_fullsized":"2023-11-01T04:16:31.187997+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:08.297413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":43827200,"num_objects":81,"num_object_clones":32,"num_object_copies":567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":2,"num_read":235,"num_read_kb":35741,"num_write":281,"num_write_kb":47072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":74031104,"data_stored":90603520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":332,"ondisk_log_size":332,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":22,"seq":94489280522,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29580,"kb_used_data":2560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342260,"statfs":{"total":96636764160,"available":96606474240,"internally_reserved":0,"allocated":2621440,"data_stored":2757974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,6,7,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82599999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1120000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.284}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]}]},{"osd":14,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31728,"kb_used_data":4708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340112,"statfs":{"total":96636764160,"available":96604274688,"internally_reserved":0,"allocated":4820992,"data_stored":5309238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,5,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2090000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]}]},{"osd":13,"up_from":22,"seq":94489280522,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32224,"kb_used_data":5204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339616,"statfs":{"total":96636764160,"available":96603766784,"internally_reserved":0,"allocated":5328896,"data_stored":6516052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.883}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.216}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.923}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7410000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.968}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3849999999999998}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4710000000000001}]}]},{"osd":6,"up_from":21,"seq":90194313227,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":4380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":4485120,"data_stored":5036856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[3,4,5,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1819999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":295.12700000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]}]},{"osd":0,"up_from":22,"seq":94489280522,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34056,"kb_used_data":7032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337784,"statfs":{"total":96636764160,"available":96601890816,"internally_reserved":0,"allocated":7200768,"data_stored":8397622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.272}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]}]},{"osd":1,"up_from":22,"seq":94489280522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":5372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":5500928,"data_stored":6349622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2210000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.093}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.28}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.335}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.395}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]}]},{"osd":2,"up_from":22,"seq":94489280523,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":5424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":5554176,"data_stored":5923638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2410000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.319}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]}]},{"osd":3,"up_from":22,"seq":94489280522,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33712,"kb_used_data":6688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338128,"statfs":{"total":96636764160,"available":96602243072,"internally_reserved":0,"allocated":6848512,"data_stored":8303414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4790000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.383}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.536}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.341}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]}]},{"osd":4,"up_from":22,"seq":94489280522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34096,"kb_used_data":7068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337744,"statfs":{"total":96636764160,"available":96601849856,"internally_reserved":0,"allocated":7237632,"data_stored":8428340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[3,5,6,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2589999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3340000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4079999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.645}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6000000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.472}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5660000000000001}]}]},{"osd":5,"up_from":22,"seq":94489280522,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":5112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":5234688,"data_stored":5788470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,6,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3360000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4370000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":154.78299999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.296}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.274}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6419999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.351}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]}]},{"osd":7,"up_from":22,"seq":94489280521,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":5480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":5611520,"data_stored":6644534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.585}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5129999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]}]},{"osd":8,"up_from":22,"seq":94489280523,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32636,"kb_used_data":5612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339204,"statfs":{"total":96636764160,"available":96603344896,"internally_reserved":0,"allocated":5746688,"data_stored":6624054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":197.03700000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]}]},{"osd":9,"up_from":22,"seq":94489280522,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30852,"kb_used_data":3828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340988,"statfs":{"total":96636764160,"available":96605171712,"internally_reserved":0,"allocated":3919872,"data_stored":4100918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[3,6,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97299999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2549999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.413}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3600000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4690000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5409999999999999}]}]},{"osd":10,"up_from":22,"seq":94489280522,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30520,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341320,"statfs":{"total":96636764160,"available":96605511680,"internally_reserved":0,"allocated":3579904,"data_stored":3904310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1919999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.357}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4279999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.506}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4750000000000001}]}]},{"osd":11,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34544,"kb_used_data":7520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337296,"statfs":{"total":96636764160,"available":96601391104,"internally_reserved":0,"allocated":7700480,"data_stored":8508214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9350000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7850000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.488}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2109999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.974}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4889999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1589999999999998}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4430000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1360000000000001}]}]},{"osd":12,"up_from":22,"seq":94489280523,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":4380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":4485120,"data_stored":4737844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.117}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]}],"pool_statfs":[{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6529024,"data_stored":8044544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4829184,"data_stored":5996544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4882432,"data_stored":5570560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6176768,"data_stored":7950336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6549504,"data_stored":8065024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4562944,"data_stored":5435392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3829760,"data_stored":4694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4939776,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":5074944,"data_stored":6270976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":3248128,"data_stored":3747840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2908160,"data_stored":3551232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":7028736,"data_stored":8155136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":3796992,"data_stored":4374528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":4108288,"data_stored":5623808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4149248,"data_stored":4956160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":1417216,"data_stored":1875968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-01T04:16:35.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 16 2023-11-01T04:16:35.794 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:35.794 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:35.800 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:35.800+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:35.808 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-01T04:16:35.895 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:35.892+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:35.932 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:35.932 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:35.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-01T04:16:35.996 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:35.994+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:36.070 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:36.070 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:36.084 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-01T04:16:36.096 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:36.096+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:36.109 INFO:teuthology.orchestra.run.smithi153.stderr:set pool 3 pgp_num to 16 2023-11-01T04:16:36.119 INFO:tasks.rados.rados.0.smithi190.stdout:166: done (1 left) 2023-11-01T04:16:36.119 INFO:tasks.rados.rados.0.smithi190.stdout:169: done (0 left) 2023-11-01T04:16:36.119 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 31 current snap is 14 2023-11-01T04:16:36.119 INFO:tasks.rados.rados.0.smithi190.stdout:170: seq_num 79 ranges {0=573440,573440=425984,999424=344064} 2023-11-01T04:16:36.123 INFO:tasks.rados.rados.0.smithi190.stdout:170: writing smithi19039307-31 from 0 to 573440 tid 1 2023-11-01T04:16:36.127 INFO:tasks.rados.rados.0.smithi190.stdout:170: writing smithi19039307-31 from 573440 to 999424 tid 2 2023-11-01T04:16:36.131 INFO:tasks.rados.rados.0.smithi190.stdout:170: writing smithi19039307-31 from 999424 to 1343488 tid 3 2023-11-01T04:16:36.131 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 16 current snap is 14 2023-11-01T04:16:36.131 INFO:tasks.rados.rados.0.smithi190.stdout:171: seq_num 80 ranges {262144=589824,851968=573440,1425408=196608} 2023-11-01T04:16:36.138 INFO:tasks.rados.rados.0.smithi190.stdout:171: writing smithi19039307-16 from 262144 to 851968 tid 1 2023-11-01T04:16:36.143 INFO:tasks.rados.rados.0.smithi190.stdout:171: writing smithi19039307-16 from 851968 to 1425408 tid 2 2023-11-01T04:16:36.145 INFO:tasks.rados.rados.0.smithi190.stdout:171: writing smithi19039307-16 from 1425408 to 1622016 tid 3 2023-11-01T04:16:36.146 INFO:tasks.rados.rados.0.smithi190.stdout:172: copy_from oid 10 from oid 39 current snap is 14 2023-11-01T04:16:36.146 INFO:tasks.rados.rados.0.smithi190.stdout:173: read oid 48 snap 2 2023-11-01T04:16:36.146 INFO:tasks.rados.rados.0.smithi190.stdout:173: expect (ObjNum 47 snap 0 seq_num 47) 2023-11-01T04:16:36.147 INFO:tasks.rados.rados.0.smithi190.stdout:174: copy_from oid 50 from oid 1 current snap is 14 2023-11-01T04:16:36.147 INFO:tasks.rados.rados.0.smithi190.stdout:175: snap_remove snap 8 2023-11-01T04:16:36.197 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:36.194+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:36.208 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:36.208 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:36.222 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-01T04:16:36.298 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:36.296+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:36.346 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:36.346 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:36.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-01T04:16:36.405 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:36.405+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:36.482 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:36.482 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:36.496 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-01T04:16:36.499 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:36.498+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:36.601 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:36.599+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:36.662 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:36.663 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:36.677 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-01T04:16:36.701 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:36.697+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:36.781 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-01T04:16:36.809 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:36.810 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:36.824 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-01T04:16:36.828 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:36.828+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:36.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-01T04:16:36.919 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:36.920+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:36.943 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:36.944 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:36.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-01T04:16:37.003 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:37.004+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:37.022 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:37.023 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-11-01T04:16:37.043 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:37.044 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-01T04:16:37.064 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:37.064 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-01T04:16:37.084 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:37.084 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2023-11-01T04:16:37.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2023-11-01T04:16:37.085 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2023-11-01T04:16:37.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.086 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 36/31 ECSubWrite(tid=13, reqid=client.4559.0:134, at_version=36'8, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:37.086 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:15.171984+0000", 2023-11-01T04:16:37.086 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.909938225000001, 2023-11-01T04:16:37.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101850327, 2023-11-01T04:16:37.087 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.087 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.087 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.088 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.088 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.171984+0000", 2023-11-01T04:16:37.088 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.089 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.089 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.089 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.171984+0000", 2023-11-01T04:16:37.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.091 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.171985+0000", 2023-11-01T04:16:37.091 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3999999999999999e-06 2023-11-01T04:16:37.091 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.172017+0000", 2023-11-01T04:16:37.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1788000000000001e-05 2023-11-01T04:16:37.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.172041+0000", 2023-11-01T04:16:37.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4153000000000001e-05 2023-11-01T04:16:37.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.272212+0000", 2023-11-01T04:16:37.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10017137399999999 2023-11-01T04:16:37.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.272363+0000", 2023-11-01T04:16:37.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015028400000000001 2023-11-01T04:16:37.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.272381+0000", 2023-11-01T04:16:37.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7893e-05 2023-11-01T04:16:37.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.273743+0000", 2023-11-01T04:16:37.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001362009 2023-11-01T04:16:37.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.099 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.273834+0000", 2023-11-01T04:16:37.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1426000000000002e-05 2023-11-01T04:16:37.100 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.100 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.101 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.101 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 36/31 ECSubWrite(tid=26, reqid=client.4559.0:142, at_version=36'13, trim_to=0'0, roll_forward_to=35'12))", 2023-11-01T04:16:37.102 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:15.204322+0000", 2023-11-01T04:16:37.102 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.877599952000001, 2023-11-01T04:16:37.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102807592, 2023-11-01T04:16:37.102 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.103 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.103 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.204322+0000", 2023-11-01T04:16:37.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.204322+0000", 2023-11-01T04:16:37.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.204324+0000", 2023-11-01T04:16:37.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9400000000000001e-06 2023-11-01T04:16:37.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.204568+0000", 2023-11-01T04:16:37.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00024425800000000002 2023-11-01T04:16:37.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.204598+0000", 2023-11-01T04:16:37.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9845e-05 2023-11-01T04:16:37.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.204614+0000", 2023-11-01T04:16:37.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5919000000000001e-05 2023-11-01T04:16:37.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.304866+0000", 2023-11-01T04:16:37.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100252465 2023-11-01T04:16:37.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.304891+0000", 2023-11-01T04:16:37.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4068999999999999e-05 2023-11-01T04:16:37.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.307052+0000", 2023-11-01T04:16:37.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021609989999999998 2023-11-01T04:16:37.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.307130+0000", 2023-11-01T04:16:37.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8096999999999994e-05 2023-11-01T04:16:37.115 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.115 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.115 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.116 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.3s2 36/31 ECSubWrite(tid=189, reqid=client.4559.0:151, at_version=36'6, trim_to=0'0, roll_forward_to=35'4))", 2023-11-01T04:16:37.116 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:15.378807+0000", 2023-11-01T04:16:37.117 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.703115351000001, 2023-11-01T04:16:37.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102355366, 2023-11-01T04:16:37.117 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.117 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.117 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.378807+0000", 2023-11-01T04:16:37.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.378807+0000", 2023-11-01T04:16:37.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.378808+0000", 2023-11-01T04:16:37.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.544e-06 2023-11-01T04:16:37.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.122 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.378973+0000", 2023-11-01T04:16:37.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016468 2023-11-01T04:16:37.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.122 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.379003+0000", 2023-11-01T04:16:37.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9998999999999999e-05 2023-11-01T04:16:37.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.479104+0000", 2023-11-01T04:16:37.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10010121399999999 2023-11-01T04:16:37.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.479245+0000", 2023-11-01T04:16:37.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000140932 2023-11-01T04:16:37.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.479261+0000", 2023-11-01T04:16:37.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5739e-05 2023-11-01T04:16:37.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.481083+0000", 2023-11-01T04:16:37.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001822735 2023-11-01T04:16:37.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.481162+0000", 2023-11-01T04:16:37.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8522999999999993e-05 2023-11-01T04:16:37.129 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.129 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.129 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.130 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 36/31 ECSubWrite(tid=16, reqid=client.4559.0:147, at_version=36'10, trim_to=0'0, roll_forward_to=36'8))", 2023-11-01T04:16:37.130 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:15.382672+0000", 2023-11-01T04:16:37.130 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.699250357, 2023-11-01T04:16:37.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102493001, 2023-11-01T04:16:37.131 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.131 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.131 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.382672+0000", 2023-11-01T04:16:37.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.133 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.133 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.133 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.382672+0000", 2023-11-01T04:16:37.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.134 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.134 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.382673+0000", 2023-11-01T04:16:37.134 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5099999999999999e-06 2023-11-01T04:16:37.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.135 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.382896+0000", 2023-11-01T04:16:37.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022287199999999999 2023-11-01T04:16:37.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.382921+0000", 2023-11-01T04:16:37.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5378999999999999e-05 2023-11-01T04:16:37.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.483022+0000", 2023-11-01T04:16:37.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100100408 2023-11-01T04:16:37.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.483128+0000", 2023-11-01T04:16:37.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000105732 2023-11-01T04:16:37.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.483155+0000", 2023-11-01T04:16:37.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7793e-05 2023-11-01T04:16:37.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.485084+0000", 2023-11-01T04:16:37.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001928777 2023-11-01T04:16:37.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.485165+0000", 2023-11-01T04:16:37.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0530000000000003e-05 2023-11-01T04:16:37.143 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.143 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.143 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.144 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 36/31 ECSubWrite(tid=29, reqid=client.4559.0:142, at_version=0'0, trim_to=0'0, roll_forward_to=36'13))", 2023-11-01T04:16:37.144 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:15.666223+0000", 2023-11-01T04:16:37.144 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.415698659, 2023-11-01T04:16:37.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10214487, 2023-11-01T04:16:37.145 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.145 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.145 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.666223+0000", 2023-11-01T04:16:37.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.666223+0000", 2023-11-01T04:16:37.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.666225+0000", 2023-11-01T04:16:37.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.933e-06 2023-11-01T04:16:37.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.666246+0000", 2023-11-01T04:16:37.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1036e-05 2023-11-01T04:16:37.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.666277+0000", 2023-11-01T04:16:37.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0392e-05 2023-11-01T04:16:37.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.766419+0000", 2023-11-01T04:16:37.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100142615 2023-11-01T04:16:37.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.766641+0000", 2023-11-01T04:16:37.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022152200000000001 2023-11-01T04:16:37.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.766667+0000", 2023-11-01T04:16:37.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6418999999999999e-05 2023-11-01T04:16:37.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.768316+0000", 2023-11-01T04:16:37.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016484480000000001 2023-11-01T04:16:37.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.768368+0000", 2023-11-01T04:16:37.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2505e-05 2023-11-01T04:16:37.156 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.157 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.157 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.158 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.3s2 36/31 ECSubRead(tid=196, to_read={3:c87735c0:::smithi19039307-50:head=0,4096,0}, subchunks={3:c87735c0:::smithi19039307-50:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:37.158 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:15.821807+0000", 2023-11-01T04:16:37.158 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.260114497, 2023-11-01T04:16:37.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20089694399999999, 2023-11-01T04:16:37.158 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.159 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.159 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.821807+0000", 2023-11-01T04:16:37.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.821807+0000", 2023-11-01T04:16:37.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.821809+0000", 2023-11-01T04:16:37.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.296e-06 2023-11-01T04:16:37.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.821825+0000", 2023-11-01T04:16:37.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6463000000000001e-05 2023-11-01T04:16:37.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.821853+0000", 2023-11-01T04:16:37.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7512e-05 2023-11-01T04:16:37.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.921962+0000", 2023-11-01T04:16:37.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100109578 2023-11-01T04:16:37.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.022264+0000", 2023-11-01T04:16:37.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100301818 2023-11-01T04:16:37.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.022704+0000", 2023-11-01T04:16:37.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00044027699999999999 2023-11-01T04:16:37.167 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.167 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.167 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.168 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 37/31 ECSubWrite(tid=22, reqid=client.4559.0:187, at_version=37'15, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:16:37.168 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.969636+0000", 2023-11-01T04:16:37.169 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.112285906, 2023-11-01T04:16:37.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19731178799999999, 2023-11-01T04:16:37.169 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.169 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.169 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969636+0000", 2023-11-01T04:16:37.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969636+0000", 2023-11-01T04:16:37.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969638+0000", 2023-11-01T04:16:37.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9199999999999998e-06 2023-11-01T04:16:37.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969695+0000", 2023-11-01T04:16:37.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7497999999999997e-05 2023-11-01T04:16:37.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969721+0000", 2023-11-01T04:16:37.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5024999999999998e-05 2023-11-01T04:16:37.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969736+0000", 2023-11-01T04:16:37.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5383000000000002e-05 2023-11-01T04:16:37.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.165457+0000", 2023-11-01T04:16:37.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19572152500000001 2023-11-01T04:16:37.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.165472+0000", 2023-11-01T04:16:37.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4713e-05 2023-11-01T04:16:37.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.166866+0000", 2023-11-01T04:16:37.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013936039999999999 2023-11-01T04:16:37.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.166948+0000", 2023-11-01T04:16:37.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2120000000000007e-05 2023-11-01T04:16:37.180 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.180 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.180 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.181 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.fs6 37/31 ECSubWrite(tid=8, reqid=client.4559.0:188, at_version=37'4, trim_to=0'0, roll_forward_to=35'2))", 2023-11-01T04:16:37.181 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.169234+0000", 2023-11-01T04:16:37.181 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.9126884, 2023-11-01T04:16:37.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10309233800000001, 2023-11-01T04:16:37.182 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.182 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.182 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.169234+0000", 2023-11-01T04:16:37.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.169234+0000", 2023-11-01T04:16:37.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.169234+0000", 2023-11-01T04:16:37.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0699999999999998e-07 2023-11-01T04:16:37.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.169374+0000", 2023-11-01T04:16:37.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000139665 2023-11-01T04:16:37.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.169403+0000", 2023-11-01T04:16:37.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8886000000000001e-05 2023-11-01T04:16:37.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.269510+0000", 2023-11-01T04:16:37.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100107338 2023-11-01T04:16:37.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.269778+0000", 2023-11-01T04:16:37.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00026818700000000003 2023-11-01T04:16:37.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.269837+0000", 2023-11-01T04:16:37.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8913999999999999e-05 2023-11-01T04:16:37.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.272242+0000", 2023-11-01T04:16:37.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002404851 2023-11-01T04:16:37.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.272326+0000", 2023-11-01T04:16:37.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4090000000000003e-05 2023-11-01T04:16:37.192 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.192 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.192 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.193 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.fs6 37/31 ECSubWrite(tid=9, reqid=client.4559.0:188, at_version=0'0, trim_to=0'0, roll_forward_to=37'4))", 2023-11-01T04:16:37.193 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.274610+0000", 2023-11-01T04:16:37.193 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.807311757000001, 2023-11-01T04:16:37.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102160717, 2023-11-01T04:16:37.194 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.194 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.194 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.274610+0000", 2023-11-01T04:16:37.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.274610+0000", 2023-11-01T04:16:37.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.274612+0000", 2023-11-01T04:16:37.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.99e-06 2023-11-01T04:16:37.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.274641+0000", 2023-11-01T04:16:37.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8921999999999999e-05 2023-11-01T04:16:37.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.274762+0000", 2023-11-01T04:16:37.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000121283 2023-11-01T04:16:37.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.374865+0000", 2023-11-01T04:16:37.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100102414 2023-11-01T04:16:37.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.375064+0000", 2023-11-01T04:16:37.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019955600000000001 2023-11-01T04:16:37.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.201 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.201 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.375090+0000", 2023-11-01T04:16:37.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5466e-05 2023-11-01T04:16:37.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.376673+0000", 2023-11-01T04:16:37.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00158358 2023-11-01T04:16:37.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.376771+0000", 2023-11-01T04:16:37.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7505999999999998e-05 2023-11-01T04:16:37.204 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.204 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.204 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.205 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.3s2 37/31 ECSubWrite(tid=208, reqid=client.4559.0:212, at_version=37'9, trim_to=0'0, roll_forward_to=36'7))", 2023-11-01T04:16:37.205 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.580549+0000", 2023-11-01T04:16:37.205 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.501373308000002, 2023-11-01T04:16:37.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103548872, 2023-11-01T04:16:37.206 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.206 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.206 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.580549+0000", 2023-11-01T04:16:37.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.580549+0000", 2023-11-01T04:16:37.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.580551+0000", 2023-11-01T04:16:37.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8929999999999999e-06 2023-11-01T04:16:37.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.580689+0000", 2023-11-01T04:16:37.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000138498 2023-11-01T04:16:37.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.580739+0000", 2023-11-01T04:16:37.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0049000000000001e-05 2023-11-01T04:16:37.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.680884+0000", 2023-11-01T04:16:37.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100145 2023-11-01T04:16:37.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.681059+0000", 2023-11-01T04:16:37.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000175181 2023-11-01T04:16:37.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.213 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.681084+0000", 2023-11-01T04:16:37.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5151999999999999e-05 2023-11-01T04:16:37.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.684027+0000", 2023-11-01T04:16:37.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002942475 2023-11-01T04:16:37.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.684098+0000", 2023-11-01T04:16:37.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0623999999999993e-05 2023-11-01T04:16:37.216 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.216 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.216 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.217 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 41/31 ECSubWrite(tid=56, reqid=client.4559.0:267, at_version=41'22, trim_to=0'0, roll_forward_to=37'19))", 2023-11-01T04:16:37.217 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:21.120537+0000", 2023-11-01T04:16:37.217 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.961384744, 2023-11-01T04:16:37.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102796897, 2023-11-01T04:16:37.217 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.218 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.218 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.218 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.120537+0000", 2023-11-01T04:16:37.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.219 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.120537+0000", 2023-11-01T04:16:37.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.120538+0000", 2023-11-01T04:16:37.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2270000000000001e-06 2023-11-01T04:16:37.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.120904+0000", 2023-11-01T04:16:37.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00036540100000000001 2023-11-01T04:16:37.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.222 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.222 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.121052+0000", 2023-11-01T04:16:37.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014824200000000001 2023-11-01T04:16:37.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.121073+0000", 2023-11-01T04:16:37.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0519000000000001e-05 2023-11-01T04:16:37.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.121204+0000", 2023-11-01T04:16:37.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013181 2023-11-01T04:16:37.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.121221+0000", 2023-11-01T04:16:37.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6449000000000001e-05 2023-11-01T04:16:37.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.226 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.226 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.223266+0000", 2023-11-01T04:16:37.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102045108 2023-11-01T04:16:37.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.226 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.226 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.227 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.223334+0000", 2023-11-01T04:16:37.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8140999999999997e-05 2023-11-01T04:16:37.227 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.227 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.227 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.228 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 41/31 ECSubWrite(tid=57, reqid=client.4559.0:268, at_version=41'23, trim_to=0'0, roll_forward_to=37'19))", 2023-11-01T04:16:37.228 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:21.121864+0000", 2023-11-01T04:16:37.228 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.960058211, 2023-11-01T04:16:37.228 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102430093, 2023-11-01T04:16:37.229 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.229 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.229 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.230 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.121864+0000", 2023-11-01T04:16:37.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.121864+0000", 2023-11-01T04:16:37.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.121865+0000", 2023-11-01T04:16:37.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6869999999999999e-06 2023-11-01T04:16:37.232 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.121937+0000", 2023-11-01T04:16:37.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1044999999999995e-05 2023-11-01T04:16:37.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.233 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.121965+0000", 2023-11-01T04:16:37.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8829000000000001e-05 2023-11-01T04:16:37.234 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.234 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.234 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.121986+0000", 2023-11-01T04:16:37.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1169e-05 2023-11-01T04:16:37.235 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.235 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.235 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.222969+0000", 2023-11-01T04:16:37.235 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10098249300000001 2023-11-01T04:16:37.236 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.236 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.236 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.236 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.222977+0000", 2023-11-01T04:16:37.236 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4120000000000001e-06 2023-11-01T04:16:37.236 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.237 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.224265+0000", 2023-11-01T04:16:37.237 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012877850000000001 2023-11-01T04:16:37.237 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.224294+0000", 2023-11-01T04:16:37.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8673000000000001e-05 2023-11-01T04:16:37.238 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.239 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.239 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.239 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 41/31 ECSubWrite(tid=58, reqid=osd.12.0:58, at_version=41'25, trim_to=0'0, roll_forward_to=37'19))", 2023-11-01T04:16:37.240 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:21.122902+0000", 2023-11-01T04:16:37.240 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.959019978000001, 2023-11-01T04:16:37.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105779886, 2023-11-01T04:16:37.240 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.240 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.240 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.241 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.122902+0000", 2023-11-01T04:16:37.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.242 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.242 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.242 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.122902+0000", 2023-11-01T04:16:37.242 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.242 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.242 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.243 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.243 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.122903+0000", 2023-11-01T04:16:37.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3370000000000001e-06 2023-11-01T04:16:37.243 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.244 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.244 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.122949+0000", 2023-11-01T04:16:37.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5649000000000003e-05 2023-11-01T04:16:37.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.244 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.245 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.245 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.122981+0000", 2023-11-01T04:16:37.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2502000000000002e-05 2023-11-01T04:16:37.245 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.245 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.223072+0000", 2023-11-01T04:16:37.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100090422 2023-11-01T04:16:37.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.223448+0000", 2023-11-01T04:16:37.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00037613399999999997 2023-11-01T04:16:37.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.247 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.223471+0000", 2023-11-01T04:16:37.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3088999999999999e-05 2023-11-01T04:16:37.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.248 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.248 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.228640+0000", 2023-11-01T04:16:37.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051686800000000001 2023-11-01T04:16:37.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.249 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.249 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.228682+0000", 2023-11-01T04:16:37.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2073000000000002e-05 2023-11-01T04:16:37.250 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.250 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.250 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.251 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s4 41/31 ECSubWrite(tid=36, reqid=osd.14.0:35, at_version=0'0, trim_to=0'0, roll_forward_to=41'14))", 2023-11-01T04:16:37.251 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:21.443284+0000", 2023-11-01T04:16:37.251 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.638638214, 2023-11-01T04:16:37.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101811875, 2023-11-01T04:16:37.251 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.252 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.252 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.443284+0000", 2023-11-01T04:16:37.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.253 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.253 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.443284+0000", 2023-11-01T04:16:37.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.254 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.254 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.443286+0000", 2023-11-01T04:16:37.254 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.823e-06 2023-11-01T04:16:37.255 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.255 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.255 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.255 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.443352+0000", 2023-11-01T04:16:37.255 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6628e-05 2023-11-01T04:16:37.255 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.443385+0000", 2023-11-01T04:16:37.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3176000000000002e-05 2023-11-01T04:16:37.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.257 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.257 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.257 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.443440+0000", 2023-11-01T04:16:37.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4718e-05 2023-11-01T04:16:37.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.258 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.443609+0000", 2023-11-01T04:16:37.258 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016845099999999999 2023-11-01T04:16:37.258 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.443638+0000", 2023-11-01T04:16:37.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9641999999999998e-05 2023-11-01T04:16:37.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.260 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.545007+0000", 2023-11-01T04:16:37.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10136847 2023-11-01T04:16:37.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.260 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.261 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.545096+0000", 2023-11-01T04:16:37.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8967000000000004e-05 2023-11-01T04:16:37.261 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.261 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.261 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.262 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 41/31 ECSubWrite(tid=38, reqid=osd.11.0:38, at_version=41'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-01T04:16:37.262 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:21.645856+0000", 2023-11-01T04:16:37.262 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.436065835000001, 2023-11-01T04:16:37.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102041799, 2023-11-01T04:16:37.263 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.263 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.263 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.264 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.645856+0000", 2023-11-01T04:16:37.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.264 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.264 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.264 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.264 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.645856+0000", 2023-11-01T04:16:37.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.265 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.645858+0000", 2023-11-01T04:16:37.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3769999999999999e-06 2023-11-01T04:16:37.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.645897+0000", 2023-11-01T04:16:37.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9654999999999999e-05 2023-11-01T04:16:37.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.267 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.267 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.645922+0000", 2023-11-01T04:16:37.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5111999999999999e-05 2023-11-01T04:16:37.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.268 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.268 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.746054+0000", 2023-11-01T04:16:37.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10013180300000001 2023-11-01T04:16:37.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.746197+0000", 2023-11-01T04:16:37.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014276100000000001 2023-11-01T04:16:37.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.270 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.746214+0000", 2023-11-01T04:16:37.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7427000000000001e-05 2023-11-01T04:16:37.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.747822+0000", 2023-11-01T04:16:37.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001607874 2023-11-01T04:16:37.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.272 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.272 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.747898+0000", 2023-11-01T04:16:37.272 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5790000000000005e-05 2023-11-01T04:16:37.272 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.272 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.272 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.273 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.273 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.273 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.3s2 47/31 ECSubWrite(tid=244, reqid=client.4559.0:359, at_version=47'14, trim_to=0'0, roll_forward_to=41'11))", 2023-11-01T04:16:37.273 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:27.511910+0000", 2023-11-01T04:16:37.273 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.570012341, 2023-11-01T04:16:37.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102941901, 2023-11-01T04:16:37.274 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.274 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.274 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.274 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.274 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.511910+0000", 2023-11-01T04:16:37.275 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.275 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.275 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.275 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.275 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.511910+0000", 2023-11-01T04:16:37.276 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.276 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.276 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.276 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.276 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.511919+0000", 2023-11-01T04:16:37.276 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6199999999999994e-06 2023-11-01T04:16:37.277 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.277 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.277 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.512087+0000", 2023-11-01T04:16:37.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016782200000000001 2023-11-01T04:16:37.277 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.278 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.278 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.278 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.512119+0000", 2023-11-01T04:16:37.278 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1761999999999998e-05 2023-11-01T04:16:37.278 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.279 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.279 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.512141+0000", 2023-11-01T04:16:37.279 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2643999999999999e-05 2023-11-01T04:16:37.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.280 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.280 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.612358+0000", 2023-11-01T04:16:37.280 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100216728 2023-11-01T04:16:37.280 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.280 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.280 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.281 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.612374+0000", 2023-11-01T04:16:37.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5773e-05 2023-11-01T04:16:37.281 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.281 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.281 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.282 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.614749+0000", 2023-11-01T04:16:37.282 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023747579999999998 2023-11-01T04:16:37.282 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.282 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.282 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.283 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.614852+0000", 2023-11-01T04:16:37.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010279400000000001 2023-11-01T04:16:37.283 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.283 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.283 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.284 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.284 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s4 48/31 ECSubWrite(tid=46, reqid=client.4559.0:373, at_version=48'23, trim_to=0'0, roll_forward_to=46'20))", 2023-11-01T04:16:37.284 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.188434+0000", 2023-11-01T04:16:37.284 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 8.8934877019999998, 2023-11-01T04:16:37.284 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102726041, 2023-11-01T04:16:37.284 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.285 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.285 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.285 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.285 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.285 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.188434+0000", 2023-11-01T04:16:37.285 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.286 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.286 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.286 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.188434+0000", 2023-11-01T04:16:37.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.188436+0000", 2023-11-01T04:16:37.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6029999999999999e-06 2023-11-01T04:16:37.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.288 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.288 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.188627+0000", 2023-11-01T04:16:37.288 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019136700000000001 2023-11-01T04:16:37.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.289 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.289 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.188655+0000", 2023-11-01T04:16:37.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7492999999999999e-05 2023-11-01T04:16:37.289 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.289 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.290 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.188671+0000", 2023-11-01T04:16:37.290 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5906e-05 2023-11-01T04:16:37.290 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.288892+0000", 2023-11-01T04:16:37.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100221435 2023-11-01T04:16:37.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.291 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.288908+0000", 2023-11-01T04:16:37.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6195999999999999e-05 2023-11-01T04:16:37.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.291079+0000", 2023-11-01T04:16:37.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002170497 2023-11-01T04:16:37.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.293 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.291160+0000", 2023-11-01T04:16:37.293 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1544000000000004e-05 2023-11-01T04:16:37.294 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.294 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.294 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.294 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.294 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.294 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s4 48/31 ECSubWrite(tid=47, reqid=client.4559.0:374, at_version=48'24, trim_to=0'0, roll_forward_to=46'20))", 2023-11-01T04:16:37.295 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.191223+0000", 2023-11-01T04:16:37.295 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 8.8906990770000007, 2023-11-01T04:16:37.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10251178499999999, 2023-11-01T04:16:37.295 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.295 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.295 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.191223+0000", 2023-11-01T04:16:37.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.297 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.297 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.191223+0000", 2023-11-01T04:16:37.297 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.297 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.298 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.298 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.191225+0000", 2023-11-01T04:16:37.298 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.61e-06 2023-11-01T04:16:37.298 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.298 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.298 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.299 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.191281+0000", 2023-11-01T04:16:37.299 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6795000000000001e-05 2023-11-01T04:16:37.299 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.299 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.299 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.299 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.191311+0000", 2023-11-01T04:16:37.300 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0015999999999999e-05 2023-11-01T04:16:37.300 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.300 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.300 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.300 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.291401+0000", 2023-11-01T04:16:37.301 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100089179 2023-11-01T04:16:37.301 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.301 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.301 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.301 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.291544+0000", 2023-11-01T04:16:37.301 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000143826 2023-11-01T04:16:37.302 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.302 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.302 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.302 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.291572+0000", 2023-11-01T04:16:37.302 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7821999999999999e-05 2023-11-01T04:16:37.302 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.303 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.303 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.303 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.293667+0000", 2023-11-01T04:16:37.303 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020943789999999999 2023-11-01T04:16:37.303 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.304 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.304 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.304 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.293735+0000", 2023-11-01T04:16:37.304 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8157999999999994e-05 2023-11-01T04:16:37.304 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.304 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.305 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.305 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.305 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.305 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.3s2 51/31 ECSubWrite(tid=255, reqid=client.4559.0:398, at_version=0'0, trim_to=0'0, roll_forward_to=51'19))", 2023-11-01T04:16:37.305 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:31.598375+0000", 2023-11-01T04:16:37.305 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.4835466540000004, 2023-11-01T04:16:37.306 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102169709, 2023-11-01T04:16:37.306 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.306 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.306 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.306 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.307 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.307 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.598375+0000", 2023-11-01T04:16:37.307 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.307 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.307 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.307 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.308 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.598375+0000", 2023-11-01T04:16:37.308 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.308 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.308 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.308 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.308 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.598377+0000", 2023-11-01T04:16:37.309 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6530000000000001e-06 2023-11-01T04:16:37.309 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.309 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.309 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.309 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.598400+0000", 2023-11-01T04:16:37.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3370000000000002e-05 2023-11-01T04:16:37.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.598431+0000", 2023-11-01T04:16:37.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0889000000000002e-05 2023-11-01T04:16:37.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.598449+0000", 2023-11-01T04:16:37.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7730000000000001e-05 2023-11-01T04:16:37.312 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.312 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.312 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.312 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.598613+0000", 2023-11-01T04:16:37.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016362200000000001 2023-11-01T04:16:37.312 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.313 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.313 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.313 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.598644+0000", 2023-11-01T04:16:37.313 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1312000000000001e-05 2023-11-01T04:16:37.313 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.313 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.314 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.314 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.700480+0000", 2023-11-01T04:16:37.314 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101836465 2023-11-01T04:16:37.314 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.314 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.315 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.700545+0000", 2023-11-01T04:16:37.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4667999999999998e-05 2023-11-01T04:16:37.315 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.315 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.315 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.316 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.316 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 53/31 ECSubWrite(tid=82, reqid=client.4559.0:405, at_version=53'37, trim_to=0'0, roll_forward_to=50'35))", 2023-11-01T04:16:37.316 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:33.313117+0000", 2023-11-01T04:16:37.316 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.7688046740000001, 2023-11-01T04:16:37.316 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104799267, 2023-11-01T04:16:37.317 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.317 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.317 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.317 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.317 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.318 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313117+0000", 2023-11-01T04:16:37.318 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.318 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.318 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313117+0000", 2023-11-01T04:16:37.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.319 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.319 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.319 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313118+0000", 2023-11-01T04:16:37.320 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9999999999999998e-07 2023-11-01T04:16:37.320 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.320 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.320 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.320 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313430+0000", 2023-11-01T04:16:37.320 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00031231199999999997 2023-11-01T04:16:37.321 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.321 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.321 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.321 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313460+0000", 2023-11-01T04:16:37.321 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9879999999999999e-05 2023-11-01T04:16:37.321 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.322 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.322 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.322 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313477+0000", 2023-11-01T04:16:37.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7186000000000001e-05 2023-11-01T04:16:37.322 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.323 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.323 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.323 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313619+0000", 2023-11-01T04:16:37.323 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014189699999999999 2023-11-01T04:16:37.323 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.323 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.324 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.324 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313648+0000", 2023-11-01T04:16:37.324 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8812000000000001e-05 2023-11-01T04:16:37.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.325 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.325 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.417809+0000", 2023-11-01T04:16:37.325 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104160975 2023-11-01T04:16:37.325 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.325 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.325 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.417917+0000", 2023-11-01T04:16:37.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107705 2023-11-01T04:16:37.326 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.326 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.326 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.326 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.327 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.327 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:37.328 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:37.328 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-01T04:16:37.329 INFO:tasks.rados.rados.0.smithi190.stdout:175: done (5 left) 2023-11-01T04:16:37.329 INFO:tasks.rados.rados.0.smithi190.stdout:176: delete oid 18 current snap is 14 2023-11-01T04:16:37.330 INFO:tasks.rados.rados.0.smithi190.stdout:170: finishing write tid 1 to smithi19039307-31 2023-11-01T04:16:37.330 INFO:tasks.rados.rados.0.smithi190.stdout:170: finishing write tid 2 to smithi19039307-31 2023-11-01T04:16:37.330 INFO:tasks.rados.rados.0.smithi190.stdout:170: finishing write tid 3 to smithi19039307-31 2023-11-01T04:16:37.330 INFO:tasks.rados.rados.0.smithi190.stdout:170: finishing write tid 4 to smithi19039307-31 2023-11-01T04:16:37.330 INFO:tasks.rados.rados.0.smithi190.stdout:171: finishing write tid 1 to smithi19039307-16 2023-11-01T04:16:37.331 INFO:tasks.rados.rados.0.smithi190.stdout:171: finishing write tid 2 to smithi19039307-16 2023-11-01T04:16:37.331 INFO:tasks.rados.rados.0.smithi190.stdout:171: finishing write tid 3 to smithi19039307-16 2023-11-01T04:16:37.331 INFO:tasks.rados.rados.0.smithi190.stdout:171: finishing write tid 4 to smithi19039307-16 2023-11-01T04:16:37.331 INFO:tasks.rados.rados.0.smithi190.stdout:171: finishing write tid 5 to smithi19039307-16 2023-11-01T04:16:37.331 INFO:tasks.rados.rados.0.smithi190.stdout:171: oid 16 updating version 0 to 44 2023-11-01T04:16:37.331 INFO:tasks.rados.rados.0.smithi190.stdout:171: oid 16 version 44 is already newer than 42 2023-11-01T04:16:37.332 INFO:tasks.rados.rados.0.smithi190.stdout:171: oid 16 version 44 is already newer than 43 2023-11-01T04:16:37.332 INFO:tasks.rados.rados.0.smithi190.stdout:171: oid 16 updating version 44 to 45 2023-11-01T04:16:37.332 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 16 v 45 (ObjNum 80 snap 14 seq_num 80) dirty exists 2023-11-01T04:16:37.332 INFO:tasks.rados.rados.0.smithi190.stdout:171: left oid 16 (ObjNum 80 snap 14 seq_num 80) 2023-11-01T04:16:37.332 INFO:tasks.rados.rados.0.smithi190.stdout:173: expect (ObjNum 47 snap 0 seq_num 47) 2023-11-01T04:16:37.333 INFO:tasks.rados.rados.0.smithi190.stdout:170: finishing write tid 5 to smithi19039307-31 2023-11-01T04:16:37.333 INFO:tasks.rados.rados.0.smithi190.stdout:170: oid 31 updating version 0 to 32 2023-11-01T04:16:37.333 INFO:tasks.rados.rados.0.smithi190.stdout:170: oid 31 version 32 is already newer than 29 2023-11-01T04:16:37.333 INFO:tasks.rados.rados.0.smithi190.stdout:170: oid 31 version 32 is already newer than 30 2023-11-01T04:16:37.333 INFO:tasks.rados.rados.0.smithi190.stdout:170: oid 31 version 32 is already newer than 31 2023-11-01T04:16:37.334 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 31 v 32 (ObjNum 79 snap 14 seq_num 79) dirty exists 2023-11-01T04:16:37.334 INFO:tasks.rados.rados.0.smithi190.stdout:170: left oid 31 (ObjNum 79 snap 14 seq_num 79) 2023-11-01T04:16:37.334 INFO:tasks.rados.rados.0.smithi190.stdout:172: finishing copy_from racing read to smithi19039307-10 2023-11-01T04:16:37.334 INFO:tasks.rados.rados.0.smithi190.stdout:174: finishing copy_from racing read to smithi19039307-50 2023-11-01T04:16:37.334 INFO:tasks.rados.rados.0.smithi190.stdout:172: finishing copy_from to smithi19039307-10 2023-11-01T04:16:37.334 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 10 v 22 (ObjNum 66 snap 6 seq_num 66) dirty exists 2023-11-01T04:16:37.335 INFO:tasks.rados.rados.0.smithi190.stdout:174: finishing copy_from to smithi19039307-50 2023-11-01T04:16:37.335 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 50 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2023-11-01T04:16:37.335 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:37.335 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-01T04:16:37.336 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-01T04:16:37.336 INFO:teuthology.orchestra.run.smithi153.stderr:noscrub is unset 2023-11-01T04:16:37.337 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:37.337 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-01T04:16:37.338 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:37.339 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-01T04:16:37.339 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:37.339 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-01T04:16:37.340 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:37.340 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-01T04:16:37.341 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:37.341 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-01T04:16:37.341 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:37.342 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-01T04:16:37.342 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:37.342 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-01T04:16:37.343 INFO:teuthology.orchestra.run.smithi153.stdout:osd.12: {} 2023-11-01T04:16:37.343 INFO:teuthology.orchestra.run.smithi153.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-01T04:16:37.344 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:37.328+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:37.354 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:37.355 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-01T04:16:37.378 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:37.378 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-01T04:16:37.404 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:37.404 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-01T04:16:37.434 INFO:tasks.rados.rados.0.smithi190.stdout:170: done (5 left) 2023-11-01T04:16:37.434 INFO:tasks.rados.rados.0.smithi190.stdout:171: done (4 left) 2023-11-01T04:16:37.434 INFO:tasks.rados.rados.0.smithi190.stdout:172: done (3 left) 2023-11-01T04:16:37.434 INFO:tasks.rados.rados.0.smithi190.stdout:173: done (2 left) 2023-11-01T04:16:37.435 INFO:tasks.rados.rados.0.smithi190.stdout:174: done (1 left) 2023-11-01T04:16:37.435 INFO:tasks.rados.rados.0.smithi190.stdout:176: done (0 left) 2023-11-01T04:16:37.435 INFO:tasks.rados.rados.0.smithi190.stdout:177: snap_remove snap 7 2023-11-01T04:16:37.468 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:37.468+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:37.474 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:37.474 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:16:37.474 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2023-11-01T04:16:37.475 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:37.488 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-01T04:16:37.532 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:37.531+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:37.606 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:37.606 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:16:37.607 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2023-11-01T04:16:37.607 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2023-11-01T04:16:37.607 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:37.622 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-01T04:16:37.632 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:37.632+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:37.732 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:37.732+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:37.788 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:37.789 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2023-11-01T04:16:37.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2023-11-01T04:16:37.789 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2023-11-01T04:16:37.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.789 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds6 36/31 ECSubWrite(tid=29, reqid=client.4559.0:142, at_version=0'0, trim_to=0'0, roll_forward_to=36'13))", 2023-11-01T04:16:37.790 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:15.666225+0000", 2023-11-01T04:16:37.790 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.11941225, 2023-11-01T04:16:37.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20176570399999999, 2023-11-01T04:16:37.790 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.790 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.791 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.666225+0000", 2023-11-01T04:16:37.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.666225+0000", 2023-11-01T04:16:37.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.666227+0000", 2023-11-01T04:16:37.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.903e-06 2023-11-01T04:16:37.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.666249+0000", 2023-11-01T04:16:37.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1719e-05 2023-11-01T04:16:37.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.666282+0000", 2023-11-01T04:16:37.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3569000000000003e-05 2023-11-01T04:16:37.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.666305+0000", 2023-11-01T04:16:37.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2826000000000001e-05 2023-11-01T04:16:37.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.668349+0000", 2023-11-01T04:16:37.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020435620000000001 2023-11-01T04:16:37.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.668365+0000", 2023-11-01T04:16:37.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5999999999999999e-05 2023-11-01T04:16:37.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.867936+0000", 2023-11-01T04:16:37.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19957143299999999 2023-11-01T04:16:37.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.867991+0000", 2023-11-01T04:16:37.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4691999999999997e-05 2023-11-01T04:16:37.800 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.800 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.800 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.801 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds6 36/31 ECSubWrite(tid=30, reqid=client.4559.0:143, at_version=36'14, trim_to=0'0, roll_forward_to=36'13))", 2023-11-01T04:16:37.801 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:15.668035+0000", 2023-11-01T04:16:37.801 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.117602332000001, 2023-11-01T04:16:37.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20063675, 2023-11-01T04:16:37.802 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.802 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.802 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.668035+0000", 2023-11-01T04:16:37.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.668035+0000", 2023-11-01T04:16:37.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.668037+0000", 2023-11-01T04:16:37.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7799999999999999e-06 2023-11-01T04:16:37.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.668146+0000", 2023-11-01T04:16:37.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000109749 2023-11-01T04:16:37.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.668176+0000", 2023-11-01T04:16:37.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9635e-05 2023-11-01T04:16:37.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.668191+0000", 2023-11-01T04:16:37.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5116999999999999e-05 2023-11-01T04:16:37.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.866638+0000", 2023-11-01T04:16:37.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.198446442 2023-11-01T04:16:37.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.866652+0000", 2023-11-01T04:16:37.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4790000000000001e-05 2023-11-01T04:16:37.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.868584+0000", 2023-11-01T04:16:37.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001931157 2023-11-01T04:16:37.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.868672+0000", 2023-11-01T04:16:37.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8079999999999997e-05 2023-11-01T04:16:37.812 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.812 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.812 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.812 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds6 36/31 ECSubWrite(tid=31, reqid=client.4559.0:144, at_version=36'15, trim_to=0'0, roll_forward_to=36'13))", 2023-11-01T04:16:37.813 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:15.668804+0000", 2023-11-01T04:16:37.813 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.116833080999999, 2023-11-01T04:16:37.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20051640400000001, 2023-11-01T04:16:37.813 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.813 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.814 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.668804+0000", 2023-11-01T04:16:37.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.668804+0000", 2023-11-01T04:16:37.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.668805+0000", 2023-11-01T04:16:37.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0589999999999999e-06 2023-11-01T04:16:37.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.668830+0000", 2023-11-01T04:16:37.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.474e-05 2023-11-01T04:16:37.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.668851+0000", 2023-11-01T04:16:37.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1058999999999999e-05 2023-11-01T04:16:37.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.668864+0000", 2023-11-01T04:16:37.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2659e-05 2023-11-01T04:16:37.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.867615+0000", 2023-11-01T04:16:37.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.198751135 2023-11-01T04:16:37.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.867632+0000", 2023-11-01T04:16:37.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7499e-05 2023-11-01T04:16:37.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.869257+0000", 2023-11-01T04:16:37.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001624786 2023-11-01T04:16:37.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.869321+0000", 2023-11-01T04:16:37.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3466999999999994e-05 2023-11-01T04:16:37.823 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.823 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.823 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.824 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(osd.12.22:4 3.1s0 3:8bb0e774:::smithi19039307-12:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:37.824 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.066507+0000", 2023-11-01T04:16:37.824 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.719130065000002, 2023-11-01T04:16:37.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.179832354, 2023-11-01T04:16:37.824 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.825 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:37.825 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.066507+0000", 2023-11-01T04:16:37.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.066507+0000", 2023-11-01T04:16:37.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.066510+0000", 2023-11-01T04:16:37.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5900000000000002e-06 2023-11-01T04:16:37.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.066521+0000", 2023-11-01T04:16:37.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.094e-05 2023-11-01T04:16:37.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.066549+0000", 2023-11-01T04:16:37.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8806000000000002e-05 2023-11-01T04:16:37.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.066572+0000", 2023-11-01T04:16:37.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2014999999999998e-05 2023-11-01T04:16:37.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.142843+0000", 2023-11-01T04:16:37.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.076271826000000001 2023-11-01T04:16:37.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:37.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.143101+0000", 2023-11-01T04:16:37.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025754400000000001 2023-11-01T04:16:37.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.246340+0000", 2023-11-01T04:16:37.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103238633 2023-11-01T04:16:37.833 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.833 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.833 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.834 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:333 3.1s0 3:8900b4ea:::smithi19039307-31:head [append 0~0] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:37.834 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:26.081831+0000", 2023-11-01T04:16:37.834 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.703806444, 2023-11-01T04:16:37.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.31092408799999999, 2023-11-01T04:16:37.834 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.835 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:37.835 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:37.835 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:37.835 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:37.835 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 333 2023-11-01T04:16:37.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.836 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.081831+0000", 2023-11-01T04:16:37.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.081831+0000", 2023-11-01T04:16:37.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.081834+0000", 2023-11-01T04:16:37.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8930000000000001e-06 2023-11-01T04:16:37.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.081845+0000", 2023-11-01T04:16:37.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1364e-05 2023-11-01T04:16:37.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.081901+0000", 2023-11-01T04:16:37.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5671000000000003e-05 2023-11-01T04:16:37.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.081933+0000", 2023-11-01T04:16:37.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2036000000000003e-05 2023-11-01T04:16:37.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.082112+0000", 2023-11-01T04:16:37.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017957800000000001 2023-11-01T04:16:37.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:37.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.082423+0000", 2023-11-01T04:16:37.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000310978 2023-11-01T04:16:37.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.084553+0000", 2023-11-01T04:16:37.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021296790000000002 2023-11-01T04:16:37.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.090281+0000", 2023-11-01T04:16:37.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057282710000000001 2023-11-01T04:16:37.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:37.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.392615+0000", 2023-11-01T04:16:37.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30233397200000001 2023-11-01T04:16:37.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.392755+0000", 2023-11-01T04:16:37.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000139646 2023-11-01T04:16:37.846 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.847 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.847 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.847 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:334 3.1s0 3:8900b4ea:::smithi19039307-31:head [rollback 4] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:37.847 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:26.081977+0000", 2023-11-01T04:16:37.848 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.703660450999999, 2023-11-01T04:16:37.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.31126151099999999, 2023-11-01T04:16:37.848 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.848 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:37.848 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:37.849 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:37.849 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:37.849 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 334 2023-11-01T04:16:37.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.849 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.081977+0000", 2023-11-01T04:16:37.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.081977+0000", 2023-11-01T04:16:37.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.081978+0000", 2023-11-01T04:16:37.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5600000000000004e-07 2023-11-01T04:16:37.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.081984+0000", 2023-11-01T04:16:37.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7939999999999997e-06 2023-11-01T04:16:37.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.081996+0000", 2023-11-01T04:16:37.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1669e-05 2023-11-01T04:16:37.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.082005+0000", 2023-11-01T04:16:37.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8959999999999992e-06 2023-11-01T04:16:37.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.085915+0000", 2023-11-01T04:16:37.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003909638 2023-11-01T04:16:37.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:37.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.086205+0000", 2023-11-01T04:16:37.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00028993500000000001 2023-11-01T04:16:37.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.087728+0000", 2023-11-01T04:16:37.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015233270000000001 2023-11-01T04:16:37.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.090283+0000", 2023-11-01T04:16:37.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002555229 2023-11-01T04:16:37.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:37.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.393145+0000", 2023-11-01T04:16:37.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30286192299999998 2023-11-01T04:16:37.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.393238+0000", 2023-11-01T04:16:37.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3244000000000004e-05 2023-11-01T04:16:37.860 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.860 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.861 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.861 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:335 3.1s0 3:8900b4ea:::smithi19039307-31:head [append 0~0] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:37.861 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:26.082073+0000", 2023-11-01T04:16:37.862 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.703563824, 2023-11-01T04:16:37.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.31173105099999998, 2023-11-01T04:16:37.862 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.862 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:37.862 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:37.863 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:37.863 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:37.863 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 335 2023-11-01T04:16:37.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.863 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.082073+0000", 2023-11-01T04:16:37.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.082073+0000", 2023-11-01T04:16:37.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.082074+0000", 2023-11-01T04:16:37.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8999999999999996e-07 2023-11-01T04:16:37.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.082078+0000", 2023-11-01T04:16:37.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6459999999999999e-06 2023-11-01T04:16:37.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.082089+0000", 2023-11-01T04:16:37.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1134e-05 2023-11-01T04:16:37.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.082097+0000", 2023-11-01T04:16:37.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6160000000000002e-06 2023-11-01T04:16:37.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.088753+0000", 2023-11-01T04:16:37.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006655205 2023-11-01T04:16:37.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:37.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.088931+0000", 2023-11-01T04:16:37.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017863800000000001 2023-11-01T04:16:37.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.089753+0000", 2023-11-01T04:16:37.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00082193700000000001 2023-11-01T04:16:37.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.090859+0000", 2023-11-01T04:16:37.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011059119999999999 2023-11-01T04:16:37.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:37.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.393579+0000", 2023-11-01T04:16:37.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30271954200000001 2023-11-01T04:16:37.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.393804+0000", 2023-11-01T04:16:37.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022573100000000001 2023-11-01T04:16:37.875 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.875 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.875 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.876 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 46/31 ECSubWriteReply(tid=36, last_complete=46'19, committed=1, applied=1))", 2023-11-01T04:16:37.876 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:26.288415+0000", 2023-11-01T04:16:37.876 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.497222117, 2023-11-01T04:16:37.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10483737899999999, 2023-11-01T04:16:37.876 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.877 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.877 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.288415+0000", 2023-11-01T04:16:37.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.288415+0000", 2023-11-01T04:16:37.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.288416+0000", 2023-11-01T04:16:37.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9999999999999997e-07 2023-11-01T04:16:37.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.288420+0000", 2023-11-01T04:16:37.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1169999999999997e-06 2023-11-01T04:16:37.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.288430+0000", 2023-11-01T04:16:37.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.024e-05 2023-11-01T04:16:37.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.388583+0000", 2023-11-01T04:16:37.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100152903 2023-11-01T04:16:37.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.393042+0000", 2023-11-01T04:16:37.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044587419999999999 2023-11-01T04:16:37.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.393252+0000", 2023-11-01T04:16:37.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021077699999999999 2023-11-01T04:16:37.884 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.884 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.885 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.885 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 46/31 ECSubWriteReply(tid=38, last_complete=46'20, committed=1, applied=1))", 2023-11-01T04:16:37.885 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:26.396760+0000", 2023-11-01T04:16:37.885 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.388877441, 2023-11-01T04:16:37.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.200491058, 2023-11-01T04:16:37.886 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.886 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.886 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.396760+0000", 2023-11-01T04:16:37.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.396760+0000", 2023-11-01T04:16:37.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.396761+0000", 2023-11-01T04:16:37.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.393e-06 2023-11-01T04:16:37.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.396772+0000", 2023-11-01T04:16:37.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0556e-05 2023-11-01T04:16:37.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.396787+0000", 2023-11-01T04:16:37.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.577e-05 2023-11-01T04:16:37.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.396797+0000", 2023-11-01T04:16:37.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9860000000000006e-06 2023-11-01T04:16:37.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.597178+0000", 2023-11-01T04:16:37.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20038024800000001 2023-11-01T04:16:37.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.597251+0000", 2023-11-01T04:16:37.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3104999999999999e-05 2023-11-01T04:16:37.893 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.894 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.894 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.894 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:344 3.1s0 3:89557db0:::smithi19039307-4:b [read 0~65536,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:37.895 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:27.031499+0000", 2023-11-01T04:16:37.895 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.754138245, 2023-11-01T04:16:37.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.208429738, 2023-11-01T04:16:37.895 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.895 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:37.895 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:37.896 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:37.896 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:37.896 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 344 2023-11-01T04:16:37.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.896 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.031499+0000", 2023-11-01T04:16:37.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.031499+0000", 2023-11-01T04:16:37.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.031502+0000", 2023-11-01T04:16:37.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.084e-06 2023-11-01T04:16:37.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.031519+0000", 2023-11-01T04:16:37.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6648999999999999e-05 2023-11-01T04:16:37.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.031552+0000", 2023-11-01T04:16:37.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3033e-05 2023-11-01T04:16:37.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.031574+0000", 2023-11-01T04:16:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2246e-05 2023-11-01T04:16:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.031720+0000", 2023-11-01T04:16:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014560199999999999 2023-11-01T04:16:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2023-11-01T04:16:37.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.031732+0000", 2023-11-01T04:16:37.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2279999999999999e-05 2023-11-01T04:16:37.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.236010+0000", 2023-11-01T04:16:37.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20427798999999999 2023-11-01T04:16:37.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:37.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.236313+0000", 2023-11-01T04:16:37.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00030311100000000001 2023-11-01T04:16:37.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.239929+0000", 2023-11-01T04:16:37.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0036157429999999998 2023-11-01T04:16:37.907 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.907 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.907 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.907 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:345 3.1s0 3:89557db0:::smithi19039307-4:b [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:37.908 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:27.031617+0000", 2023-11-01T04:16:37.908 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.754020301000001, 2023-11-01T04:16:37.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41125489599999998, 2023-11-01T04:16:37.908 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.908 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:37.909 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:37.909 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:37.909 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:37.909 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 345 2023-11-01T04:16:37.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.910 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.031617+0000", 2023-11-01T04:16:37.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.031617+0000", 2023-11-01T04:16:37.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.031618+0000", 2023-11-01T04:16:37.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5069999999999999e-06 2023-11-01T04:16:37.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.031627+0000", 2023-11-01T04:16:37.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5490000000000006e-06 2023-11-01T04:16:37.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.031653+0000", 2023-11-01T04:16:37.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5720000000000001e-05 2023-11-01T04:16:37.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.131825+0000", 2023-11-01T04:16:37.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10017213599999999 2023-11-01T04:16:37.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.131985+0000", 2023-11-01T04:16:37.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015986900000000001 2023-11-01T04:16:37.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:37.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.132008+0000", 2023-11-01T04:16:37.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3402999999999999e-05 2023-11-01T04:16:37.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.237826+0000", 2023-11-01T04:16:37.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10581789699999999 2023-11-01T04:16:37.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:37.918 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.237975+0000", 2023-11-01T04:16:37.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014861299999999999 2023-11-01T04:16:37.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.442872+0000", 2023-11-01T04:16:37.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.204897202 2023-11-01T04:16:37.920 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.920 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.920 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.920 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:346 3.1s0 3:89557db0:::smithi19039307-4:b [read 0~65536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:37.921 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:27.131866+0000", 2023-11-01T04:16:37.921 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.653771470000001, 2023-11-01T04:16:37.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.311494147, 2023-11-01T04:16:37.921 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.921 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:37.922 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:37.922 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:37.922 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:37.922 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 346 2023-11-01T04:16:37.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.922 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.131866+0000", 2023-11-01T04:16:37.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.131866+0000", 2023-11-01T04:16:37.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.131868+0000", 2023-11-01T04:16:37.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9999999999999999e-06 2023-11-01T04:16:37.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.131881+0000", 2023-11-01T04:16:37.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3236e-05 2023-11-01T04:16:37.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.131908+0000", 2023-11-01T04:16:37.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7436e-05 2023-11-01T04:16:37.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.927 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.131932+0000", 2023-11-01T04:16:37.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3689e-05 2023-11-01T04:16:37.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.132090+0000", 2023-11-01T04:16:37.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015772900000000001 2023-11-01T04:16:37.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:37.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.132112+0000", 2023-11-01T04:16:37.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2350000000000001e-05 2023-11-01T04:16:37.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.238493+0000", 2023-11-01T04:16:37.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.106380971 2023-11-01T04:16:37.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.931 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:37.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.238631+0000", 2023-11-01T04:16:37.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013804299999999999 2023-11-01T04:16:37.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.443360+0000", 2023-11-01T04:16:37.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20472869299999999 2023-11-01T04:16:37.932 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.933 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.933 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.933 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.1s0 47/31 ECSubReadReply(tid=40, attrs_read=0))", 2023-11-01T04:16:37.933 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:27.238935+0000", 2023-11-01T04:16:37.934 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.546701955, 2023-11-01T04:16:37.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20287168699999999, 2023-11-01T04:16:37.934 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.934 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.934 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.935 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.935 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.238935+0000", 2023-11-01T04:16:37.935 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.935 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.935 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.238935+0000", 2023-11-01T04:16:37.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.936 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.936 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.937 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.238936+0000", 2023-11-01T04:16:37.937 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6299999999999993e-07 2023-11-01T04:16:37.937 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.937 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.937 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.937 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.238971+0000", 2023-11-01T04:16:37.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5079000000000001e-05 2023-11-01T04:16:37.938 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.938 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.938 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.238993+0000", 2023-11-01T04:16:37.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1233000000000002e-05 2023-11-01T04:16:37.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.339149+0000", 2023-11-01T04:16:37.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10015679700000001 2023-11-01T04:16:37.940 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.940 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.940 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.940 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.441756+0000", 2023-11-01T04:16:37.940 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10260685999999999 2023-11-01T04:16:37.940 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.941 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.941 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.941 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.441807+0000", 2023-11-01T04:16:37.941 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0754999999999998e-05 2023-11-01T04:16:37.941 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.942 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.942 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.942 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.942 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:354 3.1s0 3:8900b4ea:::smithi19039307-31:head [append 0~786432] snapc c=[c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:37.942 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:27.473028+0000", 2023-11-01T04:16:37.943 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.312609648, 2023-11-01T04:16:37.943 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.21660732899999999, 2023-11-01T04:16:37.943 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.943 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:37.943 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:37.943 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:37.944 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:37.944 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 354 2023-11-01T04:16:37.944 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.944 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.944 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.945 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.945 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.473028+0000", 2023-11-01T04:16:37.945 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.945 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.945 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.945 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.946 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.473028+0000", 2023-11-01T04:16:37.946 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.946 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.946 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.946 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.947 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.473032+0000", 2023-11-01T04:16:37.947 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1699999999999999e-06 2023-11-01T04:16:37.947 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.947 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.947 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.947 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.474821+0000", 2023-11-01T04:16:37.948 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001789302 2023-11-01T04:16:37.948 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.948 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.948 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.948 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.474855+0000", 2023-11-01T04:16:37.948 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4258999999999998e-05 2023-11-01T04:16:37.949 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.949 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.949 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.949 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.474873+0000", 2023-11-01T04:16:37.949 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7963000000000001e-05 2023-11-01T04:16:37.950 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.950 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.950 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.950 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.475036+0000", 2023-11-01T04:16:37.950 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016294200000000001 2023-11-01T04:16:37.950 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.951 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.951 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:37.951 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.475332+0000", 2023-11-01T04:16:37.951 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00029604099999999998 2023-11-01T04:16:37.951 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.951 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.952 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.952 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.478851+0000", 2023-11-01T04:16:37.952 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035190690000000001 2023-11-01T04:16:37.952 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.952 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.953 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.953 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.487200+0000", 2023-11-01T04:16:37.953 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0083488239999999995 2023-11-01T04:16:37.953 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.953 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.953 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:37.954 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.689420+0000", 2023-11-01T04:16:37.954 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20221958600000001 2023-11-01T04:16:37.954 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.954 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.954 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.955 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.689635+0000", 2023-11-01T04:16:37.955 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021517300000000001 2023-11-01T04:16:37.955 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.955 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.955 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.955 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.956 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.956 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:355 3.1s0 3:8900b4ea:::smithi19039307-31:head [append 0~376832] snapc c=[c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:37.956 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:27.479364+0000", 2023-11-01T04:16:37.956 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.306273527, 2023-11-01T04:16:37.956 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.21078202700000001, 2023-11-01T04:16:37.956 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.957 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:37.957 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:37.957 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:37.957 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:37.957 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 355 2023-11-01T04:16:37.957 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.958 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.958 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.958 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.958 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.479364+0000", 2023-11-01T04:16:37.958 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.959 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.959 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.959 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.959 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.479364+0000", 2023-11-01T04:16:37.959 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.960 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.960 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.960 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.960 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.479367+0000", 2023-11-01T04:16:37.960 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9469999999999999e-06 2023-11-01T04:16:37.960 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.961 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.961 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.961 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480090+0000", 2023-11-01T04:16:37.961 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00072324200000000003 2023-11-01T04:16:37.961 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.961 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.962 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.962 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480131+0000", 2023-11-01T04:16:37.962 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0649000000000003e-05 2023-11-01T04:16:37.962 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.962 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.963 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480154+0000", 2023-11-01T04:16:37.963 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3699000000000001e-05 2023-11-01T04:16:37.963 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.963 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.964 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.481819+0000", 2023-11-01T04:16:37.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016643300000000001 2023-11-01T04:16:37.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.964 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.964 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:37.964 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.482097+0000", 2023-11-01T04:16:37.965 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027803799999999998 2023-11-01T04:16:37.965 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.965 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.965 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.965 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.483827+0000", 2023-11-01T04:16:37.965 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001729937 2023-11-01T04:16:37.966 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.966 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.966 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.966 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.487202+0000", 2023-11-01T04:16:37.966 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033756200000000002 2023-11-01T04:16:37.966 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.967 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.967 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:37.967 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.689966+0000", 2023-11-01T04:16:37.967 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20276409400000001 2023-11-01T04:16:37.967 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.967 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.968 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.968 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.690146+0000", 2023-11-01T04:16:37.968 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000179471 2023-11-01T04:16:37.968 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.968 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.969 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.969 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:356 3.1s0 3:8900b4ea:::smithi19039307-31:head [setxattr _header (56) in=7b] snapc c=[c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:37.969 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:27.480201+0000", 2023-11-01T04:16:37.969 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.305436154000001, 2023-11-01T04:16:37.970 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.210759007, 2023-11-01T04:16:37.970 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.970 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:37.970 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:37.970 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:37.970 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:37.971 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 356 2023-11-01T04:16:37.971 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.971 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.971 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.972 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.972 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480201+0000", 2023-11-01T04:16:37.972 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.972 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.972 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.972 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.973 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480201+0000", 2023-11-01T04:16:37.973 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.973 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.973 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.973 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.974 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480203+0000", 2023-11-01T04:16:37.974 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2129999999999998e-06 2023-11-01T04:16:37.974 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.974 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.974 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.974 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480211+0000", 2023-11-01T04:16:37.975 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.357e-06 2023-11-01T04:16:37.975 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.975 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.975 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.975 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480224+0000", 2023-11-01T04:16:37.975 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3409e-05 2023-11-01T04:16:37.976 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.976 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.976 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.976 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480234+0000", 2023-11-01T04:16:37.976 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4930000000000004e-06 2023-11-01T04:16:37.976 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.977 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.977 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.977 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.485355+0000", 2023-11-01T04:16:37.977 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051218030000000003 2023-11-01T04:16:37.977 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.977 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.978 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:37.978 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.485537+0000", 2023-11-01T04:16:37.978 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018136500000000001 2023-11-01T04:16:37.978 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.978 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.978 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:37.979 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.486426+0000", 2023-11-01T04:16:37.979 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00088951500000000001 2023-11-01T04:16:37.979 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.979 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.979 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:37.980 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.588049+0000", 2023-11-01T04:16:37.980 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10162265099999999 2023-11-01T04:16:37.980 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.980 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.980 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:37.980 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.690729+0000", 2023-11-01T04:16:37.981 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10268036799999999 2023-11-01T04:16:37.981 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.981 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.981 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.981 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.690960+0000", 2023-11-01T04:16:37.981 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000230833 2023-11-01T04:16:37.982 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.982 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.982 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.982 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.982 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.983 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:357 3.1s0 3:8900b4ea:::smithi19039307-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:37.983 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:27.480248+0000", 2023-11-01T04:16:37.983 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.305388775000001, 2023-11-01T04:16:37.983 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41393773900000003, 2023-11-01T04:16:37.983 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.984 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:37.984 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:37.984 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:37.984 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:37.984 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 357 2023-11-01T04:16:37.984 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.985 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.985 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.985 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.985 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480248+0000", 2023-11-01T04:16:37.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.985 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.986 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.986 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.986 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480248+0000", 2023-11-01T04:16:37.986 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.986 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.986 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.987 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.987 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480249+0000", 2023-11-01T04:16:37.987 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.16e-07 2023-11-01T04:16:37.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.987 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.988 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480253+0000", 2023-11-01T04:16:37.988 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7270000000000001e-06 2023-11-01T04:16:37.988 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.988 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.988 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:37.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480264+0000", 2023-11-01T04:16:37.989 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0693000000000001e-05 2023-11-01T04:16:37.989 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.989 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.989 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:37.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480272+0000", 2023-11-01T04:16:37.990 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0059999999999992e-06 2023-11-01T04:16:37.990 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.990 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.990 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.990 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.688898+0000", 2023-11-01T04:16:37.990 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20862633899999999 2023-11-01T04:16:37.991 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.991 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.991 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2023-11-01T04:16:37.991 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.689066+0000", 2023-11-01T04:16:37.991 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016846499999999999 2023-11-01T04:16:37.992 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.992 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.992 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:37.992 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.691994+0000", 2023-11-01T04:16:37.992 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029271100000000001 2023-11-01T04:16:37.992 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.993 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:37.993 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.692157+0000", 2023-11-01T04:16:37.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016358799999999999 2023-11-01T04:16:37.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.994 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:37.994 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.894186+0000", 2023-11-01T04:16:37.994 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20202899499999999 2023-11-01T04:16:37.994 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.994 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:37.995 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:37.995 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.995 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.995 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/31 ECSubWrite(tid=76, reqid=client.4559.0:352, at_version=47'33, trim_to=0'0, roll_forward_to=46'31))", 2023-11-01T04:16:37.995 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:27.893685+0000", 2023-11-01T04:16:37.995 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.8919522759999996, 2023-11-01T04:16:37.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104737232, 2023-11-01T04:16:37.996 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:37.996 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:37.996 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:37.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:37.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.893685+0000", 2023-11-01T04:16:37.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:37.998 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.893685+0000", 2023-11-01T04:16:37.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:37.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.998 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:37.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.893685+0000", 2023-11-01T04:16:37.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2399999999999998e-07 2023-11-01T04:16:37.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:37.999 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:37.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:37.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.894099+0000", 2023-11-01T04:16:38.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00041374400000000001 2023-11-01T04:16:38.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.000 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.894132+0000", 2023-11-01T04:16:38.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2515999999999998e-05 2023-11-01T04:16:38.001 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.001 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.001 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.001 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.894154+0000", 2023-11-01T04:16:38.001 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1837000000000001e-05 2023-11-01T04:16:38.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.002 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.002 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.002 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.994439+0000", 2023-11-01T04:16:38.002 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100284972 2023-11-01T04:16:38.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.003 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:38.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.994466+0000", 2023-11-01T04:16:38.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7186e-05 2023-11-01T04:16:38.003 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.004 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.004 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:38.004 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.998348+0000", 2023-11-01T04:16:38.004 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038818949999999998 2023-11-01T04:16:38.004 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.004 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.005 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.005 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.998422+0000", 2023-11-01T04:16:38.005 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4558e-05 2023-11-01T04:16:38.005 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.005 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.006 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.006 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.006 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.006 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s4 48/31 ECSubWrite(tid=52, reqid=client.4559.0:367, at_version=48'20, trim_to=0'0, roll_forward_to=43'18))", 2023-11-01T04:16:38.006 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.058466+0000", 2023-11-01T04:16:38.006 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.7271716109999993, 2023-11-01T04:16:38.007 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105248809, 2023-11-01T04:16:38.007 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.007 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:38.007 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.007 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.007 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.008 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.058466+0000", 2023-11-01T04:16:38.008 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.008 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.008 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.008 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.009 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.058466+0000", 2023-11-01T04:16:38.009 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.009 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.009 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.009 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.009 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.058466+0000", 2023-11-01T04:16:38.010 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8299999999999996e-07 2023-11-01T04:16:38.010 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.010 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.010 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.010 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.058622+0000", 2023-11-01T04:16:38.011 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015609599999999999 2023-11-01T04:16:38.011 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.011 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.011 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.011 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.058663+0000", 2023-11-01T04:16:38.011 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0169000000000001e-05 2023-11-01T04:16:38.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.012 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.012 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.012 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.058689+0000", 2023-11-01T04:16:38.012 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6815999999999999e-05 2023-11-01T04:16:38.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.013 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.013 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.013 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.058888+0000", 2023-11-01T04:16:38.013 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019865400000000001 2023-11-01T04:16:38.013 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.013 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.014 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:38.014 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.058916+0000", 2023-11-01T04:16:38.014 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8399999999999999e-05 2023-11-01T04:16:38.014 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.014 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.015 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:38.015 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.163625+0000", 2023-11-01T04:16:38.015 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104708123 2023-11-01T04:16:38.015 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.015 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.015 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.016 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.163714+0000", 2023-11-01T04:16:38.016 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9868000000000001e-05 2023-11-01T04:16:38.016 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.016 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.016 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.017 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.017 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.017 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s4 48/31 ECSubWrite(tid=53, reqid=client.4559.0:368, at_version=48'21, trim_to=0'0, roll_forward_to=43'18))", 2023-11-01T04:16:38.017 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.164887+0000", 2023-11-01T04:16:38.017 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.6207503509999999, 2023-11-01T04:16:38.017 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201153164, 2023-11-01T04:16:38.018 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.018 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:38.018 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.018 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.019 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.164887+0000", 2023-11-01T04:16:38.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.019 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.019 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.019 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.019 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.164887+0000", 2023-11-01T04:16:38.020 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.020 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.020 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.020 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.164890+0000", 2023-11-01T04:16:38.020 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6460000000000002e-06 2023-11-01T04:16:38.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.021 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.021 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.021 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165194+0000", 2023-11-01T04:16:38.021 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00030424800000000002 2023-11-01T04:16:38.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.022 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.022 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.022 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165227+0000", 2023-11-01T04:16:38.022 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3185999999999997e-05 2023-11-01T04:16:38.022 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.023 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.023 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.265323+0000", 2023-11-01T04:16:38.023 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10009591599999999 2023-11-01T04:16:38.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.265595+0000", 2023-11-01T04:16:38.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027192600000000003 2023-11-01T04:16:38.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.024 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:38.025 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.265614+0000", 2023-11-01T04:16:38.025 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9442000000000001e-05 2023-11-01T04:16:38.025 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.025 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.025 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:38.025 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.365956+0000", 2023-11-01T04:16:38.026 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100341862 2023-11-01T04:16:38.026 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.026 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.026 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.026 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.366040+0000", 2023-11-01T04:16:38.026 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3937999999999995e-05 2023-11-01T04:16:38.027 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.027 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.027 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.027 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.027 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.027 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:38.029 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-01T04:16:38.029 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:38.025+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:38.130 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:38.127+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:38.138 INFO:tasks.rados.rados.0.smithi190.stdout:177: done (0 left) 2023-11-01T04:16:38.139 INFO:tasks.rados.rados.0.smithi190.stdout:178: snap_create 2023-11-01T04:16:38.165 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:38.165 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:16:38.165 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2023-11-01T04:16:38.165 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:38.179 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-01T04:16:38.231 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:38.228+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:38.332 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:38.330+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:38.343 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:38.343 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:16:38.343 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2023-11-01T04:16:38.344 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2023-11-01T04:16:38.344 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:38.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-01T04:16:38.433 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:38.429+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:38.483 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:38.483 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2023-11-01T04:16:38.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2023-11-01T04:16:38.483 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2023-11-01T04:16:38.484 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.484 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:196 3.4s0 3:2f6e6eee:::smithi19039307-7:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:38.484 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.964841+0000", 2023-11-01T04:16:38.484 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.515829831000001, 2023-11-01T04:16:38.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.18028011799999999, 2023-11-01T04:16:38.485 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.485 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:38.485 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:38.485 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:38.485 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:38.486 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 196 2023-11-01T04:16:38.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.486 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964841+0000", 2023-11-01T04:16:38.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964841+0000", 2023-11-01T04:16:38.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.488 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.488 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964843+0000", 2023-11-01T04:16:38.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.83e-06 2023-11-01T04:16:38.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.489 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964857+0000", 2023-11-01T04:16:38.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3933e-05 2023-11-01T04:16:38.490 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.490 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.490 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.490 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964890+0000", 2023-11-01T04:16:38.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2981999999999997e-05 2023-11-01T04:16:38.491 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.491 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.491 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.491 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964912+0000", 2023-11-01T04:16:38.491 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1976000000000001e-05 2023-11-01T04:16:38.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.492 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.035898+0000", 2023-11-01T04:16:38.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.070985630999999993 2023-11-01T04:16:38.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.493 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:38.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.036060+0000", 2023-11-01T04:16:38.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000162122 2023-11-01T04:16:38.493 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.494 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:38.494 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.037402+0000", 2023-11-01T04:16:38.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013416750000000001 2023-11-01T04:16:38.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:38.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.043336+0000", 2023-11-01T04:16:38.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.005934328 2023-11-01T04:16:38.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:38.496 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.144926+0000", 2023-11-01T04:16:38.496 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101589578 2023-11-01T04:16:38.496 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.496 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.496 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.496 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.145121+0000", 2023-11-01T04:16:38.497 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019506300000000001 2023-11-01T04:16:38.497 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.497 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.497 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.497 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.498 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:197 3.4s0 3:2f6e6eee:::smithi19039307-7:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:38.498 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.964956+0000", 2023-11-01T04:16:38.498 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.515714365000001, 2023-11-01T04:16:38.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.37748192400000002, 2023-11-01T04:16:38.498 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.499 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:38.499 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:38.499 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:38.499 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:38.499 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 197 2023-11-01T04:16:38.499 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.500 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.500 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.500 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.500 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964956+0000", 2023-11-01T04:16:38.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.501 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.501 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964956+0000", 2023-11-01T04:16:38.501 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.502 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.502 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964957+0000", 2023-11-01T04:16:38.502 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4529999999999999e-06 2023-11-01T04:16:38.502 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.502 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.502 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964964+0000", 2023-11-01T04:16:38.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6769999999999998e-06 2023-11-01T04:16:38.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.503 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.503 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964981+0000", 2023-11-01T04:16:38.504 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6395000000000001e-05 2023-11-01T04:16:38.504 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.504 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964991+0000", 2023-11-01T04:16:38.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0020000000000001e-05 2023-11-01T04:16:38.505 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.505 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.505 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.038607+0000", 2023-11-01T04:16:38.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.073616930999999997 2023-11-01T04:16:38.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:38.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.038841+0000", 2023-11-01T04:16:38.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000233802 2023-11-01T04:16:38.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:38.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.040205+0000", 2023-11-01T04:16:38.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013635450000000001 2023-11-01T04:16:38.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.508 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:38.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.043458+0000", 2023-11-01T04:16:38.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032529569999999999 2023-11-01T04:16:38.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:38.509 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.342331+0000", 2023-11-01T04:16:38.509 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29887367100000001 2023-11-01T04:16:38.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.342438+0000", 2023-11-01T04:16:38.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000106473 2023-11-01T04:16:38.510 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.510 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.510 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.511 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:198 3.4s0 3:2f6e6eee:::smithi19039307-7:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:38.511 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.965004+0000", 2023-11-01T04:16:38.511 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.515666057000001, 2023-11-01T04:16:38.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.37915321499999999, 2023-11-01T04:16:38.511 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.512 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:38.512 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:38.512 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:38.512 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:38.512 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 198 2023-11-01T04:16:38.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.513 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965004+0000", 2023-11-01T04:16:38.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.514 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.514 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965004+0000", 2023-11-01T04:16:38.514 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.514 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965005+0000", 2023-11-01T04:16:38.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8599999999999997e-07 2023-11-01T04:16:38.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965011+0000", 2023-11-01T04:16:38.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1970000000000001e-06 2023-11-01T04:16:38.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965117+0000", 2023-11-01T04:16:38.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000105919 2023-11-01T04:16:38.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965150+0000", 2023-11-01T04:16:38.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2468999999999997e-05 2023-11-01T04:16:38.518 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.518 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.518 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.041140+0000", 2023-11-01T04:16:38.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.075989950000000001 2023-11-01T04:16:38.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:38.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.041378+0000", 2023-11-01T04:16:38.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023841499999999999 2023-11-01T04:16:38.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:38.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.042565+0000", 2023-11-01T04:16:38.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011867480000000001 2023-11-01T04:16:38.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:38.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.145152+0000", 2023-11-01T04:16:38.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102587356 2023-11-01T04:16:38.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.521 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:38.522 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.344048+0000", 2023-11-01T04:16:38.522 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19889615599999999 2023-11-01T04:16:38.522 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.522 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.344158+0000", 2023-11-01T04:16:38.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000109219 2023-11-01T04:16:38.523 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.523 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.523 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.524 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(osd.13.22:2 3.4s0 3:2c54b66d:::smithi19039307-30:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:38.524 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.067952+0000", 2023-11-01T04:16:38.524 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.412718759000001, 2023-11-01T04:16:38.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.27583712700000002, 2023-11-01T04:16:38.524 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.525 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:38.525 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.067952+0000", 2023-11-01T04:16:38.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.526 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.526 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.067952+0000", 2023-11-01T04:16:38.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.527 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.527 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.067955+0000", 2023-11-01T04:16:38.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2059999999999999e-06 2023-11-01T04:16:38.527 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.528 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.067967+0000", 2023-11-01T04:16:38.528 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2386e-05 2023-11-01T04:16:38.528 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.528 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.528 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.529 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.068008+0000", 2023-11-01T04:16:38.529 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0541999999999999e-05 2023-11-01T04:16:38.529 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.529 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.529 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.529 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.068030+0000", 2023-11-01T04:16:38.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2422999999999999e-05 2023-11-01T04:16:38.530 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.530 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.530 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.530 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.146030+0000", 2023-11-01T04:16:38.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.077999657 2023-11-01T04:16:38.531 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.531 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:38.531 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.146177+0000", 2023-11-01T04:16:38.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000147014 2023-11-01T04:16:38.531 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.343789+0000", 2023-11-01T04:16:38.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19761189900000001 2023-11-01T04:16:38.532 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.532 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.533 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.533 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 41/31 ECSubWriteReply(tid=36, last_complete=41'16, committed=1, applied=1))", 2023-11-01T04:16:38.533 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:21.141112+0000", 2023-11-01T04:16:38.533 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 17.339558267000001, 2023-11-01T04:16:38.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.192908777, 2023-11-01T04:16:38.534 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.534 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:38.534 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.534 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.534 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.535 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.141112+0000", 2023-11-01T04:16:38.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.535 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.535 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.535 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.141112+0000", 2023-11-01T04:16:38.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.141114+0000", 2023-11-01T04:16:38.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7430000000000001e-06 2023-11-01T04:16:38.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.141124+0000", 2023-11-01T04:16:38.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9359999999999994e-06 2023-11-01T04:16:38.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.538 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.538 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.538 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.141152+0000", 2023-11-01T04:16:38.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8238999999999998e-05 2023-11-01T04:16:38.538 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.539 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.539 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.539 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.241247+0000", 2023-11-01T04:16:38.539 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10009470199999999 2023-11-01T04:16:38.539 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.539 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.540 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.333986+0000", 2023-11-01T04:16:38.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.092739007999999998 2023-11-01T04:16:38.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.334021+0000", 2023-11-01T04:16:38.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5148999999999998e-05 2023-11-01T04:16:38.541 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.541 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.541 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.542 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 41/31 ECSubWriteReply(tid=35, last_complete=41'10, committed=1, applied=1))", 2023-11-01T04:16:38.542 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:21.241302+0000", 2023-11-01T04:16:38.542 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 17.239368548000002, 2023-11-01T04:16:38.542 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.194023956, 2023-11-01T04:16:38.542 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.543 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:38.543 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.241302+0000", 2023-11-01T04:16:38.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.544 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.544 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.544 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.544 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.241302+0000", 2023-11-01T04:16:38.544 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.544 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.545 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.545 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.545 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.241302+0000", 2023-11-01T04:16:38.545 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7999999999999995e-07 2023-11-01T04:16:38.545 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.545 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.241308+0000", 2023-11-01T04:16:38.546 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8459999999999996e-06 2023-11-01T04:16:38.546 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.241320+0000", 2023-11-01T04:16:38.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1695000000000001e-05 2023-11-01T04:16:38.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.548 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.341409+0000", 2023-11-01T04:16:38.548 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100088688 2023-11-01T04:16:38.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.548 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.548 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.434345+0000", 2023-11-01T04:16:38.549 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.092936439999999995 2023-11-01T04:16:38.549 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.549 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.549 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.549 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.435326+0000", 2023-11-01T04:16:38.549 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00098060700000000009 2023-11-01T04:16:38.550 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.550 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.550 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.550 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.550 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 43/31 ECSubWriteReply(tid=41, last_complete=43'18, committed=1, applied=1))", 2023-11-01T04:16:38.551 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:23.117024+0000", 2023-11-01T04:16:38.551 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.363646621000001, 2023-11-01T04:16:38.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19997076699999999, 2023-11-01T04:16:38.551 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.551 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:38.551 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.552 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.552 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:23.117024+0000", 2023-11-01T04:16:38.552 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.552 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.552 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.553 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.553 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:23.117024+0000", 2023-11-01T04:16:38.553 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.553 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.553 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.553 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:23.117026+0000", 2023-11-01T04:16:38.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.933e-06 2023-11-01T04:16:38.554 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:23.117036+0000", 2023-11-01T04:16:38.555 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0607e-05 2023-11-01T04:16:38.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.555 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.555 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.555 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:23.117068+0000", 2023-11-01T04:16:38.555 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1891999999999999e-05 2023-11-01T04:16:38.556 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.556 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.556 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.556 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:23.117089+0000", 2023-11-01T04:16:38.556 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1216000000000002e-05 2023-11-01T04:16:38.556 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.557 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.557 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.557 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:23.316961+0000", 2023-11-01T04:16:38.557 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19987197400000001 2023-11-01T04:16:38.557 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.558 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.558 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.558 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:23.316994+0000", 2023-11-01T04:16:38.558 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3145000000000002e-05 2023-11-01T04:16:38.558 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.559 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.559 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.559 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.559 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.559 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s1 46/31 ECSubWrite(tid=26, reqid=client.4559.0:326, at_version=46'17, trim_to=0'0, roll_forward_to=43'14))", 2023-11-01T04:16:38.559 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:26.162592+0000", 2023-11-01T04:16:38.560 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 12.31807879, 2023-11-01T04:16:38.560 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.205084134, 2023-11-01T04:16:38.560 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.560 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:38.560 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.560 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.561 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.561 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.162592+0000", 2023-11-01T04:16:38.561 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.561 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.561 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.562 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.562 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.162592+0000", 2023-11-01T04:16:38.562 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.562 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.562 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.562 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.162593+0000", 2023-11-01T04:16:38.563 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.934e-06 2023-11-01T04:16:38.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.563 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.563 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.162694+0000", 2023-11-01T04:16:38.564 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010064300000000001 2023-11-01T04:16:38.564 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.564 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.564 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.564 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.162726+0000", 2023-11-01T04:16:38.565 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1958999999999997e-05 2023-11-01T04:16:38.565 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.565 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.565 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.565 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.262885+0000", 2023-11-01T04:16:38.565 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100158707 2023-11-01T04:16:38.566 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.566 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.566 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.566 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.363201+0000", 2023-11-01T04:16:38.566 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100316563 2023-11-01T04:16:38.566 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.567 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.567 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:38.567 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.363219+0000", 2023-11-01T04:16:38.567 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7235999999999999e-05 2023-11-01T04:16:38.567 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.567 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.568 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:38.568 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.367625+0000", 2023-11-01T04:16:38.568 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044067630000000002 2023-11-01T04:16:38.568 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.568 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.568 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.569 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:26.367676+0000", 2023-11-01T04:16:38.569 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0328999999999999e-05 2023-11-01T04:16:38.569 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.569 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.569 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.569 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.570 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.570 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:367 3.4s0 3:22b97aa6:::smithi19039307-42:head [append 0~425984] snapc d=[d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:16:38.570 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.055375+0000", 2023-11-01T04:16:38.570 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.425295475, 2023-11-01T04:16:38.570 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11157336700000001, 2023-11-01T04:16:38.570 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.571 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:38.571 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:38.571 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:38.571 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:38.571 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 367 2023-11-01T04:16:38.571 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.572 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.572 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.572 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.572 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.055375+0000", 2023-11-01T04:16:38.572 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.572 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.573 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.573 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.573 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.055375+0000", 2023-11-01T04:16:38.573 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.573 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.573 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.574 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.574 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.055378+0000", 2023-11-01T04:16:38.574 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.253e-06 2023-11-01T04:16:38.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.574 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.575 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.575 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.056165+0000", 2023-11-01T04:16:38.575 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00078682800000000005 2023-11-01T04:16:38.575 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.575 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.576 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.576 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.056201+0000", 2023-11-01T04:16:38.576 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5735999999999998e-05 2023-11-01T04:16:38.576 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.576 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.576 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.577 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.056224+0000", 2023-11-01T04:16:38.577 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3589000000000001e-05 2023-11-01T04:16:38.577 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.577 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.577 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.577 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.056390+0000", 2023-11-01T04:16:38.577 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016565899999999999 2023-11-01T04:16:38.578 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.578 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.578 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:38.578 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.056572+0000", 2023-11-01T04:16:38.578 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018221099999999999 2023-11-01T04:16:38.579 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.579 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.579 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:38.579 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.058516+0000", 2023-11-01T04:16:38.579 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001944227 2023-11-01T04:16:38.579 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.580 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.580 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:38.580 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.162019+0000", 2023-11-01T04:16:38.580 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103502794 2023-11-01T04:16:38.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.580 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.581 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:38.581 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.166818+0000", 2023-11-01T04:16:38.581 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0047989570000000004 2023-11-01T04:16:38.581 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.581 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.581 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.582 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.166948+0000", 2023-11-01T04:16:38.582 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013011300000000001 2023-11-01T04:16:38.582 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.582 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.582 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.582 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.583 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:368 3.4s0 3:22b97aa6:::smithi19039307-42:head [append 0~573440] snapc d=[d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:16:38.583 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.062184+0000", 2023-11-01T04:16:38.583 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.418486678000001, 2023-11-01T04:16:38.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.305717142, 2023-11-01T04:16:38.583 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.584 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:38.584 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:38.584 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:38.584 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:38.584 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 368 2023-11-01T04:16:38.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.585 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.062184+0000", 2023-11-01T04:16:38.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.586 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.586 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.586 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.586 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.062184+0000", 2023-11-01T04:16:38.586 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.586 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.587 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.587 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.587 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.062186+0000", 2023-11-01T04:16:38.587 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7199999999999998e-06 2023-11-01T04:16:38.587 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.587 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.588 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.588 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.063184+0000", 2023-11-01T04:16:38.588 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00099736599999999996 2023-11-01T04:16:38.588 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.588 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.588 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.589 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.063229+0000", 2023-11-01T04:16:38.589 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5584999999999998e-05 2023-11-01T04:16:38.589 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.589 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.589 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.590 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.063253+0000", 2023-11-01T04:16:38.590 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3943000000000001e-05 2023-11-01T04:16:38.590 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.590 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.590 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.590 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.162390+0000", 2023-11-01T04:16:38.591 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099136827999999996 2023-11-01T04:16:38.591 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.591 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.591 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:38.591 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.162690+0000", 2023-11-01T04:16:38.591 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000300026 2023-11-01T04:16:38.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.592 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.592 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:38.592 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.164813+0000", 2023-11-01T04:16:38.592 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021232170000000002 2023-11-01T04:16:38.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.593 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.593 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:38.593 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.170806+0000", 2023-11-01T04:16:38.593 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059924710000000001 2023-11-01T04:16:38.593 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.593 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.594 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:38.594 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.367777+0000", 2023-11-01T04:16:38.594 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.1969707 2023-11-01T04:16:38.594 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.594 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.594 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.595 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.367901+0000", 2023-11-01T04:16:38.595 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012428600000000001 2023-11-01T04:16:38.595 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.595 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.595 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.595 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:369 3.4s0 3:22b97aa6:::smithi19039307-42:head [append 0~196608] snapc d=[d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:16:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.064665+0000", 2023-11-01T04:16:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.416005820000001, 2023-11-01T04:16:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40249887899999998, 2023-11-01T04:16:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 369 2023-11-01T04:16:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.064665+0000", 2023-11-01T04:16:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.064665+0000", 2023-11-01T04:16:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.064666+0000", 2023-11-01T04:16:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.64e-06 2023-11-01T04:16:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.601 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.601 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.065011+0000", 2023-11-01T04:16:38.601 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034451099999999998 2023-11-01T04:16:38.601 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.601 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.602 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.065036+0000", 2023-11-01T04:16:38.602 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5148999999999999e-05 2023-11-01T04:16:38.602 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.602 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.602 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.602 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165152+0000", 2023-11-01T04:16:38.603 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100115738 2023-11-01T04:16:38.603 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.603 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.603 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.603 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.167023+0000", 2023-11-01T04:16:38.603 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001871075 2023-11-01T04:16:38.604 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.604 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.604 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:38.604 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.167186+0000", 2023-11-01T04:16:38.604 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016346199999999999 2023-11-01T04:16:38.604 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.605 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:38.605 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.168223+0000", 2023-11-01T04:16:38.605 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010366119999999999 2023-11-01T04:16:38.605 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.606 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:38.606 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.270989+0000", 2023-11-01T04:16:38.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102765841 2023-11-01T04:16:38.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.606 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.606 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:38.607 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.466957+0000", 2023-11-01T04:16:38.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.195968693 2023-11-01T04:16:38.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.607 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.607 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.607 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.467163+0000", 2023-11-01T04:16:38.608 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020615799999999999 2023-11-01T04:16:38.608 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.608 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.608 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.608 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.609 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:370 3.4s0 3:22b97aa6:::smithi19039307-42:head [setxattr _header (56) in=7b] snapc d=[d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:16:38.609 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.165200+0000", 2023-11-01T04:16:38.609 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.315470754, 2023-11-01T04:16:38.609 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40210787999999997, 2023-11-01T04:16:38.609 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.609 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:38.610 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:38.610 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:38.610 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:38.610 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 370 2023-11-01T04:16:38.610 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.610 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.611 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.611 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165200+0000", 2023-11-01T04:16:38.611 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.612 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.612 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165200+0000", 2023-11-01T04:16:38.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.612 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.612 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.613 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165202+0000", 2023-11-01T04:16:38.613 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6000000000000001e-06 2023-11-01T04:16:38.613 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.613 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.613 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.613 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165215+0000", 2023-11-01T04:16:38.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2904e-05 2023-11-01T04:16:38.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.614 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165249+0000", 2023-11-01T04:16:38.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3655000000000002e-05 2023-11-01T04:16:38.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165268+0000", 2023-11-01T04:16:38.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8810000000000001e-05 2023-11-01T04:16:38.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.169297+0000", 2023-11-01T04:16:38.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040292280000000001 2023-11-01T04:16:38.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.617 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:38.617 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.169474+0000", 2023-11-01T04:16:38.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017749500000000001 2023-11-01T04:16:38.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.617 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:38.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.170331+0000", 2023-11-01T04:16:38.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00085666300000000002 2023-11-01T04:16:38.618 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:38.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.271064+0000", 2023-11-01T04:16:38.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10073283099999999 2023-11-01T04:16:38.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:38.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.567166+0000", 2023-11-01T04:16:38.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29610207100000002 2023-11-01T04:16:38.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.620 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.620 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.620 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.567307+0000", 2023-11-01T04:16:38.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014162300000000001 2023-11-01T04:16:38.620 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.621 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.621 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.621 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:371 3.4s0 3:22b97aa6:::smithi19039307-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:16:38.621 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.165296+0000", 2023-11-01T04:16:38.622 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.315374675999999, 2023-11-01T04:16:38.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60529993999999998, 2023-11-01T04:16:38.622 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.622 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:38.622 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:38.623 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:38.623 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:38.623 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 371 2023-11-01T04:16:38.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.623 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165296+0000", 2023-11-01T04:16:38.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165296+0000", 2023-11-01T04:16:38.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165297+0000", 2023-11-01T04:16:38.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4729999999999999e-06 2023-11-01T04:16:38.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165304+0000", 2023-11-01T04:16:38.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7730000000000002e-06 2023-11-01T04:16:38.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165338+0000", 2023-11-01T04:16:38.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.392e-05 2023-11-01T04:16:38.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165353+0000", 2023-11-01T04:16:38.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4922e-05 2023-11-01T04:16:38.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.266752+0000", 2023-11-01T04:16:38.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101399533 2023-11-01T04:16:38.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2023-11-01T04:16:38.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.266921+0000", 2023-11-01T04:16:38.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016895199999999999 2023-11-01T04:16:38.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.630 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.568170+0000", 2023-11-01T04:16:38.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30124910799999999 2023-11-01T04:16:38.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:38.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.568436+0000", 2023-11-01T04:16:38.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00026584599999999998 2023-11-01T04:16:38.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.770596+0000", 2023-11-01T04:16:38.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20215941300000001 2023-11-01T04:16:38.632 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.632 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.633 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.633 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 48/31 ECSubWriteReply(tid=55, last_complete=48'23, committed=1, applied=1))", 2023-11-01T04:16:38.633 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.172256+0000", 2023-11-01T04:16:38.633 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.308414841999999, 2023-11-01T04:16:38.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19501249100000001, 2023-11-01T04:16:38.634 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.634 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:38.634 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.635 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.172256+0000", 2023-11-01T04:16:38.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.635 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.635 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.172256+0000", 2023-11-01T04:16:38.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.636 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.636 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.636 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.636 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.172256+0000", 2023-11-01T04:16:38.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5000000000000001e-07 2023-11-01T04:16:38.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.172262+0000", 2023-11-01T04:16:38.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1529999999999996e-06 2023-11-01T04:16:38.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.172279+0000", 2023-11-01T04:16:38.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7583000000000001e-05 2023-11-01T04:16:38.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.639 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.172295+0000", 2023-11-01T04:16:38.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5376e-05 2023-11-01T04:16:38.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.367243+0000", 2023-11-01T04:16:38.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.194948119 2023-11-01T04:16:38.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.367268+0000", 2023-11-01T04:16:38.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.531e-05 2023-11-01T04:16:38.641 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.641 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.641 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.641 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.642 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 48/31 ECSubWriteReply(tid=54, last_complete=48'22, committed=1, applied=1))", 2023-11-01T04:16:38.642 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.172757+0000", 2023-11-01T04:16:38.642 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.307913586, 2023-11-01T04:16:38.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19485240200000001, 2023-11-01T04:16:38.642 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.642 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:38.643 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.172757+0000", 2023-11-01T04:16:38.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.172757+0000", 2023-11-01T04:16:38.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.172758+0000", 2023-11-01T04:16:38.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1030000000000001e-06 2023-11-01T04:16:38.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.172766+0000", 2023-11-01T04:16:38.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0499999999999992e-06 2023-11-01T04:16:38.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.172784+0000", 2023-11-01T04:16:38.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8116000000000001e-05 2023-11-01T04:16:38.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.272875+0000", 2023-11-01T04:16:38.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100091313 2023-11-01T04:16:38.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.367588+0000", 2023-11-01T04:16:38.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.094712370000000004 2023-11-01T04:16:38.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.367609+0000", 2023-11-01T04:16:38.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.145e-05 2023-11-01T04:16:38.650 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.650 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.650 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.651 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s5 51/31 ECSubWrite(tid=47, reqid=client.4559.0:394, at_version=0'0, trim_to=0'0, roll_forward_to=51'37))", 2023-11-01T04:16:38.651 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:31.783669+0000", 2023-11-01T04:16:38.651 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.6970011710000001, 2023-11-01T04:16:38.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20184263799999999, 2023-11-01T04:16:38.651 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.651 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:38.652 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.783669+0000", 2023-11-01T04:16:38.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.783669+0000", 2023-11-01T04:16:38.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.783671+0000", 2023-11-01T04:16:38.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1399999999999998e-06 2023-11-01T04:16:38.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.655 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.783694+0000", 2023-11-01T04:16:38.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3142999999999998e-05 2023-11-01T04:16:38.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.783725+0000", 2023-11-01T04:16:38.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0688999999999997e-05 2023-11-01T04:16:38.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.883871+0000", 2023-11-01T04:16:38.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100145423 2023-11-01T04:16:38.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.984170+0000", 2023-11-01T04:16:38.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10029957 2023-11-01T04:16:38.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:38.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.984187+0000", 2023-11-01T04:16:38.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6762999999999999e-05 2023-11-01T04:16:38.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:38.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.985440+0000", 2023-11-01T04:16:38.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012528649999999999 2023-11-01T04:16:38.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:31.985512+0000", 2023-11-01T04:16:38.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2045000000000006e-05 2023-11-01T04:16:38.660 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.660 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.661 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.661 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:402 3.4s0 3:22b97aa6:::smithi19039307-42:head [read 0~1900544,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2023-11-01T04:16:38.661 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:33.099768+0000", 2023-11-01T04:16:38.661 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.3809023409999996, 2023-11-01T04:16:38.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30675417700000002, 2023-11-01T04:16:38.662 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.662 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:38.662 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:38.662 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:38.663 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:38.663 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 402 2023-11-01T04:16:38.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.663 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.099768+0000", 2023-11-01T04:16:38.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.099768+0000", 2023-11-01T04:16:38.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.099771+0000", 2023-11-01T04:16:38.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.757e-06 2023-11-01T04:16:38.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.099784+0000", 2023-11-01T04:16:38.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3456e-05 2023-11-01T04:16:38.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.099818+0000", 2023-11-01T04:16:38.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3535999999999999e-05 2023-11-01T04:16:38.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.099835+0000", 2023-11-01T04:16:38.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6988999999999999e-05 2023-11-01T04:16:38.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200113+0000", 2023-11-01T04:16:38.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10027828900000001 2023-11-01T04:16:38.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2023-11-01T04:16:38.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200126+0000", 2023-11-01T04:16:38.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3122000000000001e-05 2023-11-01T04:16:38.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200695+0000", 2023-11-01T04:16:38.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00056910299999999995 2023-11-01T04:16:38.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:38.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200880+0000", 2023-11-01T04:16:38.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000184413 2023-11-01T04:16:38.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.406522+0000", 2023-11-01T04:16:38.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.205642512 2023-11-01T04:16:38.672 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.672 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.672 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.673 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:403 3.4s0 3:22b97aa6:::smithi19039307-42:head [read 0~1900544,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2023-11-01T04:16:38.673 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:33.099874+0000", 2023-11-01T04:16:38.673 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.3807965900000001, 2023-11-01T04:16:38.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30840983300000002, 2023-11-01T04:16:38.674 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.674 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:38.674 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:38.674 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:38.674 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:38.675 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 403 2023-11-01T04:16:38.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.675 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.099874+0000", 2023-11-01T04:16:38.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.099874+0000", 2023-11-01T04:16:38.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.099875+0000", 2023-11-01T04:16:38.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.23e-07 2023-11-01T04:16:38.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.099880+0000", 2023-11-01T04:16:38.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6799999999999998e-06 2023-11-01T04:16:38.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.099900+0000", 2023-11-01T04:16:38.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9786e-05 2023-11-01T04:16:38.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200087+0000", 2023-11-01T04:16:38.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100186517 2023-11-01T04:16:38.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.201507+0000", 2023-11-01T04:16:38.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014206150000000001 2023-11-01T04:16:38.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:38.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.201757+0000", 2023-11-01T04:16:38.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000250244 2023-11-01T04:16:38.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.408284+0000", 2023-11-01T04:16:38.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20652616800000001 2023-11-01T04:16:38.683 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.683 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.683 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.683 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:404 3.4s0 3:22b97aa6:::smithi19039307-42:head [read 0~1900544,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2023-11-01T04:16:38.684 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:33.200147+0000", 2023-11-01T04:16:38.684 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.2805231920000004, 2023-11-01T04:16:38.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40700329400000002, 2023-11-01T04:16:38.684 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.684 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:38.685 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:38.685 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:38.685 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:38.685 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 404 2023-11-01T04:16:38.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.685 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200147+0000", 2023-11-01T04:16:38.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200147+0000", 2023-11-01T04:16:38.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200149+0000", 2023-11-01T04:16:38.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.187e-06 2023-11-01T04:16:38.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200162+0000", 2023-11-01T04:16:38.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2943e-05 2023-11-01T04:16:38.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200189+0000", 2023-11-01T04:16:38.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7056e-05 2023-11-01T04:16:38.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200209+0000", 2023-11-01T04:16:38.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9802000000000001e-05 2023-11-01T04:16:38.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.403941+0000", 2023-11-01T04:16:38.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20373233099999999 2023-11-01T04:16:38.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:38.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.404113+0000", 2023-11-01T04:16:38.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000171855 2023-11-01T04:16:38.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.607150+0000", 2023-11-01T04:16:38.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20303711999999999 2023-11-01T04:16:38.693 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.693 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.693 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.694 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.4s0 52/31 ECSubReadReply(tid=60, attrs_read=0))", 2023-11-01T04:16:38.694 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:33.205250+0000", 2023-11-01T04:16:38.694 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.2754204209999997, 2023-11-01T04:16:38.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19859510999999999, 2023-11-01T04:16:38.695 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.695 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:38.695 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.205250+0000", 2023-11-01T04:16:38.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.205250+0000", 2023-11-01T04:16:38.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.205251+0000", 2023-11-01T04:16:38.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1999999999999998e-07 2023-11-01T04:16:38.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.205879+0000", 2023-11-01T04:16:38.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00062786099999999996 2023-11-01T04:16:38.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.205896+0000", 2023-11-01T04:16:38.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7672999999999998e-05 2023-11-01T04:16:38.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:38.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.205906+0000", 2023-11-01T04:16:38.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2499999999999995e-06 2023-11-01T04:16:38.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:38.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.403774+0000", 2023-11-01T04:16:38.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.197867985 2023-11-01T04:16:38.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:38.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.403845+0000", 2023-11-01T04:16:38.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1421000000000006e-05 2023-11-01T04:16:38.702 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.702 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.702 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.702 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.702 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.703 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:38.704 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-01T04:16:38.704 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:38.703+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:38.805 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:38.804+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:38.881 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:38.882 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2023-11-01T04:16:38.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.882 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 58/31 ECSubWriteReply(tid=66, last_complete=58'38, committed=1, applied=1))", 2023-11-01T04:16:38.882 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:38.857911+0000", 2023-11-01T04:16:38.883 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.024673915000000001, 2023-11-01T04:16:38.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.024694548, 2023-11-01T04:16:38.883 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:38.883 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "no flag points reached", 2023-11-01T04:16:38.883 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:38.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:38.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:38.857911+0000", 2023-11-01T04:16:38.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:38.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:38.857911+0000", 2023-11-01T04:16:38.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:38.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:38.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:38.857912+0000", 2023-11-01T04:16:38.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4300000000000002e-07 2023-11-01T04:16:38.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:38.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:38.857918+0000", 2023-11-01T04:16:38.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9630000000000003e-06 2023-11-01T04:16:38.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:38.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:38.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:38.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:38.857930+0000", 2023-11-01T04:16:38.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2414e-05 2023-11-01T04:16:38.888 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.889 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:38.889 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.889 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:38.889 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2023-11-01T04:16:38.889 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 1 2023-11-01T04:16:38.890 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:38.896 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-01T04:16:38.906 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:38.905+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:39.007 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:39.006+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:39.016 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:39.017 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:16:39.017 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2023-11-01T04:16:39.017 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2023-11-01T04:16:39.017 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:39.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-01T04:16:39.108 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:39.106+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:39.141 INFO:tasks.rados.rados.0.smithi190.stdout:178: done (0 left) 2023-11-01T04:16:39.141 INFO:tasks.rados.rados.0.smithi190.stdout:179: read oid 46 snap -1 2023-11-01T04:16:39.141 INFO:tasks.rados.rados.0.smithi190.stdout:179: expect (ObjNum 45 snap 0 seq_num 45) 2023-11-01T04:16:39.142 INFO:tasks.rados.rados.0.smithi190.stdout:180: read oid 7 snap 13 2023-11-01T04:16:39.142 INFO:tasks.rados.rados.0.smithi190.stdout:180: expect (ObjNum 6 snap 0 seq_num 6) 2023-11-01T04:16:39.142 INFO:tasks.rados.rados.0.smithi190.stdout:181: snap_create 2023-11-01T04:16:39.154 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:39.154 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2023-11-01T04:16:39.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2023-11-01T04:16:39.154 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2023-11-01T04:16:39.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.155 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(osd.13.22:1 3.6s0 3:6b2f2983:::smithi19039307-44:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:39.155 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.940531+0000", 2023-11-01T04:16:39.155 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.209821723000001, 2023-11-01T04:16:39.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30767406400000002, 2023-11-01T04:16:39.155 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.156 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:39.156 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940531+0000", 2023-11-01T04:16:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940531+0000", 2023-11-01T04:16:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940532+0000", 2023-11-01T04:16:39.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.86e-06 2023-11-01T04:16:39.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940541+0000", 2023-11-01T04:16:39.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7260000000000008e-06 2023-11-01T04:16:39.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940560+0000", 2023-11-01T04:16:39.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8647e-05 2023-11-01T04:16:39.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940571+0000", 2023-11-01T04:16:39.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1688999999999999e-05 2023-11-01T04:16:39.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940733+0000", 2023-11-01T04:16:39.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016122700000000001 2023-11-01T04:16:39.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2023-11-01T04:16:39.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940754+0000", 2023-11-01T04:16:39.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.16e-05 2023-11-01T04:16:39.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.943338+0000", 2023-11-01T04:16:39.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025839080000000002 2023-11-01T04:16:39.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:39.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.943489+0000", 2023-11-01T04:16:39.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151268 2023-11-01T04:16:39.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.248205+0000", 2023-11-01T04:16:39.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30471513900000002 2023-11-01T04:16:39.166 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.166 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.167 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.167 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.6s0 36/31 ECSubReadReply(tid=24, attrs_read=0))", 2023-11-01T04:16:39.167 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.945649+0000", 2023-11-01T04:16:39.167 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.204703429999999, 2023-11-01T04:16:39.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201734625, 2023-11-01T04:16:39.168 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.168 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:39.168 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.945649+0000", 2023-11-01T04:16:39.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.945649+0000", 2023-11-01T04:16:39.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.945650+0000", 2023-11-01T04:16:39.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6360000000000001e-06 2023-11-01T04:16:39.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.946159+0000", 2023-11-01T04:16:39.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00050888500000000004 2023-11-01T04:16:39.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.946183+0000", 2023-11-01T04:16:39.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3583000000000001e-05 2023-11-01T04:16:39.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.946194+0000", 2023-11-01T04:16:39.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1399e-05 2023-11-01T04:16:39.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.147265+0000", 2023-11-01T04:16:39.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20107088000000001 2023-11-01T04:16:39.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.147383+0000", 2023-11-01T04:16:39.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000118242 2023-11-01T04:16:39.175 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.176 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.176 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.176 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:199 3.7s0 3:ec8ceea5:::smithi19039307-25:2 [read 0~409600,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:39.176 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.964921+0000", 2023-11-01T04:16:39.177 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.185431262000002, 2023-11-01T04:16:39.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.2062832, 2023-11-01T04:16:39.177 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.177 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:39.177 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:39.178 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:39.178 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.178 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 199 2023-11-01T04:16:39.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.178 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964921+0000", 2023-11-01T04:16:39.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964921+0000", 2023-11-01T04:16:39.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964924+0000", 2023-11-01T04:16:39.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.593e-06 2023-11-01T04:16:39.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964938+0000", 2023-11-01T04:16:39.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4287000000000001e-05 2023-11-01T04:16:39.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964970+0000", 2023-11-01T04:16:39.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2091999999999997e-05 2023-11-01T04:16:39.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.964987+0000", 2023-11-01T04:16:39.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6855999999999999e-05 2023-11-01T04:16:39.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.065231+0000", 2023-11-01T04:16:39.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10024369700000001 2023-11-01T04:16:39.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:39.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.065522+0000", 2023-11-01T04:16:39.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00029172999999999997 2023-11-01T04:16:39.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.171204+0000", 2023-11-01T04:16:39.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105681945 2023-11-01T04:16:39.186 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.186 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.187 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.187 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:200 3.7s0 3:ec8ceea5:::smithi19039307-25:2 [read 0~409600,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:39.187 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.965027+0000", 2023-11-01T04:16:39.187 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.185325662, 2023-11-01T04:16:39.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.207316629, 2023-11-01T04:16:39.188 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.188 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:39.188 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:39.188 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:39.189 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.189 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 200 2023-11-01T04:16:39.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.189 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965027+0000", 2023-11-01T04:16:39.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965027+0000", 2023-11-01T04:16:39.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965028+0000", 2023-11-01T04:16:39.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4869999999999999e-06 2023-11-01T04:16:39.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965036+0000", 2023-11-01T04:16:39.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8720000000000002e-06 2023-11-01T04:16:39.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965061+0000", 2023-11-01T04:16:39.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4598999999999999e-05 2023-11-01T04:16:39.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965077+0000", 2023-11-01T04:16:39.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6053999999999999e-05 2023-11-01T04:16:39.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.066413+0000", 2023-11-01T04:16:39.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10133632099999999 2023-11-01T04:16:39.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:39.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.066578+0000", 2023-11-01T04:16:39.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016469100000000001 2023-11-01T04:16:39.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.172343+0000", 2023-11-01T04:16:39.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105765605 2023-11-01T04:16:39.197 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.197 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.197 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.198 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:201 3.7s0 3:ec8ceea5:::smithi19039307-25:2 [read 0~409600,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:39.198 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.965129+0000", 2023-11-01T04:16:39.198 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.185223445999998, 2023-11-01T04:16:39.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30806539799999999, 2023-11-01T04:16:39.198 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.199 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:39.199 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:39.199 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:39.199 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.199 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 201 2023-11-01T04:16:39.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.200 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965129+0000", 2023-11-01T04:16:39.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.201 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.201 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965129+0000", 2023-11-01T04:16:39.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965130+0000", 2023-11-01T04:16:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.387e-06 2023-11-01T04:16:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965138+0000", 2023-11-01T04:16:39.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8189999999999992e-06 2023-11-01T04:16:39.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965177+0000", 2023-11-01T04:16:39.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8946000000000003e-05 2023-11-01T04:16:39.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965193+0000", 2023-11-01T04:16:39.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5739e-05 2023-11-01T04:16:39.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.067121+0000", 2023-11-01T04:16:39.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101928121 2023-11-01T04:16:39.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:39.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.067291+0000", 2023-11-01T04:16:39.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016974000000000001 2023-11-01T04:16:39.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.273194+0000", 2023-11-01T04:16:39.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.205903646 2023-11-01T04:16:39.207 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.208 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.208 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.208 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.6s0 37/31 ECSubReadReply(tid=24, attrs_read=0))", 2023-11-01T04:16:39.208 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.045775+0000", 2023-11-01T04:16:39.209 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.104577145, 2023-11-01T04:16:39.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201805491, 2023-11-01T04:16:39.209 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.209 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:39.209 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.045775+0000", 2023-11-01T04:16:39.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.045775+0000", 2023-11-01T04:16:39.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.045777+0000", 2023-11-01T04:16:39.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6300000000000001e-06 2023-11-01T04:16:39.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.046433+0000", 2023-11-01T04:16:39.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00065657599999999997 2023-11-01T04:16:39.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.213 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.046462+0000", 2023-11-01T04:16:39.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8639000000000001e-05 2023-11-01T04:16:39.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.046481+0000", 2023-11-01T04:16:39.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9035999999999999e-05 2023-11-01T04:16:39.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.247513+0000", 2023-11-01T04:16:39.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20103245 2023-11-01T04:16:39.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.247581+0000", 2023-11-01T04:16:39.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7160000000000001e-05 2023-11-01T04:16:39.216 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.216 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.216 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.217 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:214 3.7s0 3:e43a5132:::smithi19039307-40:head [append 0~0] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2023-11-01T04:16:39.217 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.953956+0000", 2023-11-01T04:16:39.217 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.196396493000002, 2023-11-01T04:16:39.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.310752374, 2023-11-01T04:16:39.218 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.218 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:39.218 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:39.218 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:39.218 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.218 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 214 2023-11-01T04:16:39.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.219 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.219 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.953956+0000", 2023-11-01T04:16:39.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.953956+0000", 2023-11-01T04:16:39.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.953959+0000", 2023-11-01T04:16:39.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2890000000000002e-06 2023-11-01T04:16:39.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.222 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.222 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.953978+0000", 2023-11-01T04:16:39.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8519999999999999e-05 2023-11-01T04:16:39.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.222 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954023+0000", 2023-11-01T04:16:39.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5640999999999999e-05 2023-11-01T04:16:39.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954048+0000", 2023-11-01T04:16:39.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4569000000000001e-05 2023-11-01T04:16:39.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954206+0000", 2023-11-01T04:16:39.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015823200000000001 2023-11-01T04:16:39.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2023-11-01T04:16:39.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954219+0000", 2023-11-01T04:16:39.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3052e-05 2023-11-01T04:16:39.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.226 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.226 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.226 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.055038+0000", 2023-11-01T04:16:39.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10081902299999999 2023-11-01T04:16:39.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:39.227 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.055200+0000", 2023-11-01T04:16:39.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000162304 2023-11-01T04:16:39.227 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.228 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:39.228 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.056576+0000", 2023-11-01T04:16:39.228 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013755169999999999 2023-11-01T04:16:39.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:39.229 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063225+0000", 2023-11-01T04:16:39.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066492729999999998 2023-11-01T04:16:39.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:39.230 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.264608+0000", 2023-11-01T04:16:39.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201382965 2023-11-01T04:16:39.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.264708+0000", 2023-11-01T04:16:39.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9988999999999994e-05 2023-11-01T04:16:39.231 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.231 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.231 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.232 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:215 3.7s0 3:e43a5132:::smithi19039307-40:head [rollback 3] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2023-11-01T04:16:39.232 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.954091+0000", 2023-11-01T04:16:39.232 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.196261117999999, 2023-11-01T04:16:39.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.31100268599999997, 2023-11-01T04:16:39.232 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.233 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:39.233 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:39.233 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:39.233 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.233 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215 2023-11-01T04:16:39.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.234 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.234 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.234 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954091+0000", 2023-11-01T04:16:39.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.234 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.235 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.235 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954091+0000", 2023-11-01T04:16:39.235 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.235 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.236 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.236 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954092+0000", 2023-11-01T04:16:39.236 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.33e-07 2023-11-01T04:16:39.236 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.236 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.236 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954097+0000", 2023-11-01T04:16:39.237 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3770000000000004e-06 2023-11-01T04:16:39.237 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.237 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954111+0000", 2023-11-01T04:16:39.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3145999999999999e-05 2023-11-01T04:16:39.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954123+0000", 2023-11-01T04:16:39.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2296000000000001e-05 2023-11-01T04:16:39.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.239 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.239 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954349+0000", 2023-11-01T04:16:39.239 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022646499999999999 2023-11-01T04:16:39.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.240 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:39.240 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954362+0000", 2023-11-01T04:16:39.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.312e-05 2023-11-01T04:16:39.240 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.241 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.057871+0000", 2023-11-01T04:16:39.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10350849299999999 2023-11-01T04:16:39.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:39.242 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.058109+0000", 2023-11-01T04:16:39.242 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000237631 2023-11-01T04:16:39.242 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.242 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.242 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:39.242 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.059576+0000", 2023-11-01T04:16:39.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001467154 2023-11-01T04:16:39.243 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.243 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:39.243 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063386+0000", 2023-11-01T04:16:39.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038099010000000001 2023-11-01T04:16:39.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.244 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.244 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:39.244 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.265007+0000", 2023-11-01T04:16:39.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20162109 2023-11-01T04:16:39.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.245 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.245 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.265094+0000", 2023-11-01T04:16:39.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.708e-05 2023-11-01T04:16:39.245 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.246 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.246 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.246 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:351 3.6s0 3:6b2f2983:::smithi19039307-44:head [delete] snapc c=[c,b,4] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:39.246 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:27.245720+0000", 2023-11-01T04:16:39.247 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.904632297999999, 2023-11-01T04:16:39.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20551417499999999, 2023-11-01T04:16:39.247 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.247 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:39.247 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:39.248 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:39.248 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.248 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 351 2023-11-01T04:16:39.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.248 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.249 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.249 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.245720+0000", 2023-11-01T04:16:39.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.245720+0000", 2023-11-01T04:16:39.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.245723+0000", 2023-11-01T04:16:39.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6369999999999999e-06 2023-11-01T04:16:39.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.245733+0000", 2023-11-01T04:16:39.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9329999999999996e-06 2023-11-01T04:16:39.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.245765+0000", 2023-11-01T04:16:39.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2088999999999997e-05 2023-11-01T04:16:39.253 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.253 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.245785+0000", 2023-11-01T04:16:39.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0585999999999999e-05 2023-11-01T04:16:39.253 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.254 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.254 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.245959+0000", 2023-11-01T04:16:39.254 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017419500000000001 2023-11-01T04:16:39.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.255 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:39.255 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.246249+0000", 2023-11-01T04:16:39.255 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000289752 2023-11-01T04:16:39.255 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.255 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.255 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:39.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.248116+0000", 2023-11-01T04:16:39.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001866913 2023-11-01T04:16:39.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:39.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.250427+0000", 2023-11-01T04:16:39.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023108069999999998 2023-11-01T04:16:39.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.257 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.257 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:39.257 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.451125+0000", 2023-11-01T04:16:39.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20069830899999999 2023-11-01T04:16:39.258 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.258 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.451234+0000", 2023-11-01T04:16:39.258 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108954 2023-11-01T04:16:39.258 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.259 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.259 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.259 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:372 3.7s0 3:ec8ceea5:::smithi19039307-25:head [append 0~442368] snapc d=[d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:16:39.259 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.071141+0000", 2023-11-01T04:16:39.260 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.079211697, 2023-11-01T04:16:39.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.323732619, 2023-11-01T04:16:39.260 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.260 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:39.260 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:39.261 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:39.261 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.261 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 372 2023-11-01T04:16:39.261 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.261 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.262 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.071141+0000", 2023-11-01T04:16:39.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.071141+0000", 2023-11-01T04:16:39.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.071143+0000", 2023-11-01T04:16:39.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8669999999999998e-06 2023-11-01T04:16:39.264 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.264 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.264 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.264 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.071960+0000", 2023-11-01T04:16:39.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081667000000000003 2023-11-01T04:16:39.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.265 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.071994+0000", 2023-11-01T04:16:39.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4363000000000003e-05 2023-11-01T04:16:39.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.072011+0000", 2023-11-01T04:16:39.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6350000000000001e-05 2023-11-01T04:16:39.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.267 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.267 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.072142+0000", 2023-11-01T04:16:39.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013155900000000001 2023-11-01T04:16:39.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.268 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2023-11-01T04:16:39.268 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.072155+0000", 2023-11-01T04:16:39.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2393e-05 2023-11-01T04:16:39.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.268 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.181494+0000", 2023-11-01T04:16:39.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.109339383 2023-11-01T04:16:39.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:39.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.181655+0000", 2023-11-01T04:16:39.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016068500000000001 2023-11-01T04:16:39.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:39.270 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.183750+0000", 2023-11-01T04:16:39.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020950859999999999 2023-11-01T04:16:39.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:39.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.192077+0000", 2023-11-01T04:16:39.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0083268979999999992 2023-11-01T04:16:39.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.272 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:39.272 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.394749+0000", 2023-11-01T04:16:39.272 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.202671929 2023-11-01T04:16:39.272 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.273 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.273 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.394873+0000", 2023-11-01T04:16:39.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012443600000000001 2023-11-01T04:16:39.273 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.273 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.273 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.274 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.274 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:373 3.7s0 3:ec8ceea5:::smithi19039307-25:head [append 0~720896] snapc d=[d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:16:39.274 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.079379+0000", 2023-11-01T04:16:39.274 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.070973548, 2023-11-01T04:16:39.274 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.61637890200000001, 2023-11-01T04:16:39.275 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.275 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:39.275 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:39.275 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:39.275 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.275 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 373 2023-11-01T04:16:39.276 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.276 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.276 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.276 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.276 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.079379+0000", 2023-11-01T04:16:39.276 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.277 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.277 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.277 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.079379+0000", 2023-11-01T04:16:39.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.278 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.278 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.278 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.278 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.079382+0000", 2023-11-01T04:16:39.278 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8830000000000002e-06 2023-11-01T04:16:39.278 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.279 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.279 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080658+0000", 2023-11-01T04:16:39.279 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001276461 2023-11-01T04:16:39.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.280 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.280 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080691+0000", 2023-11-01T04:16:39.280 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2793000000000003e-05 2023-11-01T04:16:39.280 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.280 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.280 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.281 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080707+0000", 2023-11-01T04:16:39.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6113e-05 2023-11-01T04:16:39.281 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.281 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.281 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.281 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.180945+0000", 2023-11-01T04:16:39.282 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100237957 2023-11-01T04:16:39.282 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.282 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.282 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:39.282 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.180960+0000", 2023-11-01T04:16:39.282 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5207e-05 2023-11-01T04:16:39.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.283 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.283 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.283 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.185900+0000", 2023-11-01T04:16:39.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049396860000000004 2023-11-01T04:16:39.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.284 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.284 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:39.284 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.186250+0000", 2023-11-01T04:16:39.284 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034973099999999999 2023-11-01T04:16:39.284 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.284 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.285 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:39.285 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.188479+0000", 2023-11-01T04:16:39.285 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022291809999999998 2023-11-01T04:16:39.285 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.285 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.285 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:39.285 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.192289+0000", 2023-11-01T04:16:39.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038101939999999998 2023-11-01T04:16:39.286 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.286 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:39.286 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.695644+0000", 2023-11-01T04:16:39.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50335487199999995 2023-11-01T04:16:39.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.695758+0000", 2023-11-01T04:16:39.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000113824 2023-11-01T04:16:39.288 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.288 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.288 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.289 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:374 3.7s0 3:ec8ceea5:::smithi19039307-25:head [append 0~49152] snapc d=[d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:16:39.289 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.080763+0000", 2023-11-01T04:16:39.289 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.069589513, 2023-11-01T04:16:39.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.61527303499999997, 2023-11-01T04:16:39.289 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.289 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:39.290 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:39.290 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:39.290 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.290 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 374 2023-11-01T04:16:39.290 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.291 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.291 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080763+0000", 2023-11-01T04:16:39.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080763+0000", 2023-11-01T04:16:39.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.293 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080764+0000", 2023-11-01T04:16:39.293 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7090000000000001e-06 2023-11-01T04:16:39.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.294 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080824+0000", 2023-11-01T04:16:39.294 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9429000000000003e-05 2023-11-01T04:16:39.294 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.294 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.294 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080839+0000", 2023-11-01T04:16:39.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5048999999999999e-05 2023-11-01T04:16:39.295 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.295 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080850+0000", 2023-11-01T04:16:39.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.097e-05 2023-11-01T04:16:39.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.181081+0000", 2023-11-01T04:16:39.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100230598 2023-11-01T04:16:39.297 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.297 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:39.297 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.181092+0000", 2023-11-01T04:16:39.297 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.152e-05 2023-11-01T04:16:39.297 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.298 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.298 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.298 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.190068+0000", 2023-11-01T04:16:39.298 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.008975841 2023-11-01T04:16:39.298 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.298 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.299 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:39.299 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.190248+0000", 2023-11-01T04:16:39.299 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000180318 2023-11-01T04:16:39.299 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.299 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.299 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:39.300 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.191149+0000", 2023-11-01T04:16:39.300 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00090082499999999995 2023-11-01T04:16:39.300 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.300 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.300 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:39.300 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.194083+0000", 2023-11-01T04:16:39.301 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029342859999999999 2023-11-01T04:16:39.301 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.301 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.301 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:39.301 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.695930+0000", 2023-11-01T04:16:39.301 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50184695999999995 2023-11-01T04:16:39.302 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.302 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.302 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.302 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.696036+0000", 2023-11-01T04:16:39.302 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010553 2023-11-01T04:16:39.302 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.303 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.303 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.303 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.303 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.303 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:375 3.7s0 3:ec8ceea5:::smithi19039307-25:head [setxattr _header (56) in=7b] snapc d=[d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:16:39.303 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.080873+0000", 2023-11-01T04:16:39.304 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.069479713, 2023-11-01T04:16:39.304 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.61552053200000001, 2023-11-01T04:16:39.304 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.304 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:39.304 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:39.304 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:39.305 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.305 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 375 2023-11-01T04:16:39.305 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.305 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.305 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.305 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.306 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080873+0000", 2023-11-01T04:16:39.306 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.306 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.306 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.306 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.306 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080873+0000", 2023-11-01T04:16:39.307 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.307 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.307 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.307 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.307 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080874+0000", 2023-11-01T04:16:39.307 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9999999999999995e-07 2023-11-01T04:16:39.308 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.308 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.308 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.308 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080879+0000", 2023-11-01T04:16:39.308 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5099999999999998e-06 2023-11-01T04:16:39.308 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.309 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.309 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.309 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080889+0000", 2023-11-01T04:16:39.309 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0176e-05 2023-11-01T04:16:39.309 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080901+0000", 2023-11-01T04:16:39.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1273e-05 2023-11-01T04:16:39.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.192640+0000", 2023-11-01T04:16:39.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11173952600000001 2023-11-01T04:16:39.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:39.312 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.192811+0000", 2023-11-01T04:16:39.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017141500000000001 2023-11-01T04:16:39.312 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.312 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.312 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:39.312 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.193529+0000", 2023-11-01T04:16:39.313 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00071754999999999996 2023-11-01T04:16:39.313 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.313 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.313 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:39.313 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.194745+0000", 2023-11-01T04:16:39.313 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001215783 2023-11-01T04:16:39.314 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.314 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.314 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:39.314 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.696253+0000", 2023-11-01T04:16:39.314 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50150775000000003 2023-11-01T04:16:39.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.315 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.315 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.696393+0000", 2023-11-01T04:16:39.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014054900000000001 2023-11-01T04:16:39.315 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.316 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.316 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.316 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.316 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:376 3.7s0 3:ec8ceea5:::smithi19039307-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:16:39.316 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.080912+0000", 2023-11-01T04:16:39.317 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.069440653999999, 2023-11-01T04:16:39.317 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.91925732000000004, 2023-11-01T04:16:39.317 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.317 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:16:39.317 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:39.317 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:39.318 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.318 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 376 2023-11-01T04:16:39.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.318 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.319 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080912+0000", 2023-11-01T04:16:39.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.319 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.319 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.319 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.320 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080912+0000", 2023-11-01T04:16:39.320 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.320 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.320 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.320 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.320 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080912+0000", 2023-11-01T04:16:39.321 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9299999999999997e-07 2023-11-01T04:16:39.321 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.321 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.321 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.321 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080918+0000", 2023-11-01T04:16:39.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4229999999999996e-06 2023-11-01T04:16:39.322 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.322 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.322 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.322 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080931+0000", 2023-11-01T04:16:39.323 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3474e-05 2023-11-01T04:16:39.323 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.323 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.323 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.323 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080940+0000", 2023-11-01T04:16:39.323 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2859999999999999e-06 2023-11-01T04:16:39.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.324 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.324 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.194154+0000", 2023-11-01T04:16:39.324 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.113214551 2023-11-01T04:16:39.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.325 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.325 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2023-11-01T04:16:39.325 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.194295+0000", 2023-11-01T04:16:39.325 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014040299999999999 2023-11-01T04:16:39.325 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.325 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.326 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.697223+0000", 2023-11-01T04:16:39.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50292827699999998 2023-11-01T04:16:39.326 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.326 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.327 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:39.327 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.697478+0000", 2023-11-01T04:16:39.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000255116 2023-11-01T04:16:39.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:29.000169+0000", 2023-11-01T04:16:39.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30269099700000002 2023-11-01T04:16:39.330 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.330 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.330 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.331 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 48/31 ECSubWriteReply(tid=46, last_complete=48'23, committed=1, applied=1))", 2023-11-01T04:16:39.331 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.394573+0000", 2023-11-01T04:16:39.331 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.755779293, 2023-11-01T04:16:39.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30069670300000001, 2023-11-01T04:16:39.331 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.331 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:39.332 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.394573+0000", 2023-11-01T04:16:39.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.394573+0000", 2023-11-01T04:16:39.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.394574+0000", 2023-11-01T04:16:39.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5000000000000002e-07 2023-11-01T04:16:39.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.394578+0000", 2023-11-01T04:16:39.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.87e-06 2023-11-01T04:16:39.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.394595+0000", 2023-11-01T04:16:39.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6483000000000001e-05 2023-11-01T04:16:39.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.494750+0000", 2023-11-01T04:16:39.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10015468700000001 2023-11-01T04:16:39.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.695237+0000", 2023-11-01T04:16:39.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20048759699999999 2023-11-01T04:16:39.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.695270+0000", 2023-11-01T04:16:39.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2416000000000003e-05 2023-11-01T04:16:39.338 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.339 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.339 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.339 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:410 3.7s0 3:ec8ceea5:::smithi19039307-25:head [append 0~507904] snapc 12=[12,11,10,f,e,d,b,4] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-11-01T04:16:39.339 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:34.208836+0000", 2023-11-01T04:16:39.340 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.9415162749999997, 2023-11-01T04:16:39.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41852388099999999, 2023-11-01T04:16:39.340 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.340 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:39.340 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:39.340 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:39.341 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.341 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 410 2023-11-01T04:16:39.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.341 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.208836+0000", 2023-11-01T04:16:39.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.208836+0000", 2023-11-01T04:16:39.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.208842+0000", 2023-11-01T04:16:39.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9499999999999998e-06 2023-11-01T04:16:39.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.344 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.210301+0000", 2023-11-01T04:16:39.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001459012 2023-11-01T04:16:39.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.345 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.345 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.345 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.210335+0000", 2023-11-01T04:16:39.345 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.392e-05 2023-11-01T04:16:39.345 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.345 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.346 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.346 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.210362+0000", 2023-11-01T04:16:39.346 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6971999999999999e-05 2023-11-01T04:16:39.346 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.346 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.347 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.347 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.311425+0000", 2023-11-01T04:16:39.347 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10106277299999999 2023-11-01T04:16:39.347 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.347 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.348 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:39.348 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.311594+0000", 2023-11-01T04:16:39.348 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016940299999999999 2023-11-01T04:16:39.348 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.348 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.348 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:39.349 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.313619+0000", 2023-11-01T04:16:39.349 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020251380000000001 2023-11-01T04:16:39.349 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.349 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.349 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:39.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.421318+0000", 2023-11-01T04:16:39.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10769884 2023-11-01T04:16:39.350 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:39.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.627230+0000", 2023-11-01T04:16:39.351 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20591240599999999 2023-11-01T04:16:39.351 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.351 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.351 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.627360+0000", 2023-11-01T04:16:39.351 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000129467 2023-11-01T04:16:39.352 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.352 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.352 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.352 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.352 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.352 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:411 3.7s0 3:ec8ceea5:::smithi19039307-25:head [append 0~507904] snapc 12=[12,11,10,f,e,d,b,4] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-11-01T04:16:39.353 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:34.210408+0000", 2023-11-01T04:16:39.353 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.9399442850000002, 2023-11-01T04:16:39.353 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.51680641900000002, 2023-11-01T04:16:39.353 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.353 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:39.353 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:39.354 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:39.354 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.354 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 411 2023-11-01T04:16:39.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.354 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.355 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.355 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.210408+0000", 2023-11-01T04:16:39.355 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.355 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.355 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.355 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.210408+0000", 2023-11-01T04:16:39.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.356 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.210410+0000", 2023-11-01T04:16:39.357 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1799999999999999e-06 2023-11-01T04:16:39.357 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.357 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.357 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.357 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.211993+0000", 2023-11-01T04:16:39.357 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001582955 2023-11-01T04:16:39.358 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.358 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.358 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.358 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212047+0000", 2023-11-01T04:16:39.358 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4175999999999998e-05 2023-11-01T04:16:39.359 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.359 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.359 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.359 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212085+0000", 2023-11-01T04:16:39.359 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7969000000000002e-05 2023-11-01T04:16:39.359 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.360 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.360 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.360 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.417049+0000", 2023-11-01T04:16:39.360 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20496331200000001 2023-11-01T04:16:39.360 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.360 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.361 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:39.361 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.417369+0000", 2023-11-01T04:16:39.361 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032029900000000002 2023-11-01T04:16:39.361 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.361 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.361 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:39.362 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.419383+0000", 2023-11-01T04:16:39.362 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020145789999999998 2023-11-01T04:16:39.362 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.362 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.362 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:39.363 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.426157+0000", 2023-11-01T04:16:39.363 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0067739690000000003 2023-11-01T04:16:39.363 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.363 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.363 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:39.363 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.727103+0000", 2023-11-01T04:16:39.364 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30094544000000001 2023-11-01T04:16:39.364 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.364 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.364 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.364 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.727214+0000", 2023-11-01T04:16:39.364 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011154000000000001 2023-11-01T04:16:39.365 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.365 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.365 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.365 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.365 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.365 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:412 3.7s0 3:ec8ceea5:::smithi19039307-25:head [append 0~262144] snapc 12=[12,11,10,f,e,d,b,4] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-11-01T04:16:39.366 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:34.212131+0000", 2023-11-01T04:16:39.366 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.9382209030000004, 2023-11-01T04:16:39.366 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.51556788200000003, 2023-11-01T04:16:39.366 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.366 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:39.366 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:39.367 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:39.367 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.367 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 412 2023-11-01T04:16:39.367 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.367 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.367 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.368 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.368 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212131+0000", 2023-11-01T04:16:39.368 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.368 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.368 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.369 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212131+0000", 2023-11-01T04:16:39.369 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.369 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.369 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.369 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.369 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212134+0000", 2023-11-01T04:16:39.370 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4930000000000001e-06 2023-11-01T04:16:39.370 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.370 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.370 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.370 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212623+0000", 2023-11-01T04:16:39.370 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00048915199999999997 2023-11-01T04:16:39.371 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.371 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.371 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.371 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212653+0000", 2023-11-01T04:16:39.371 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9703e-05 2023-11-01T04:16:39.371 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.372 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212675+0000", 2023-11-01T04:16:39.372 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2116e-05 2023-11-01T04:16:39.372 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.372 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.421675+0000", 2023-11-01T04:16:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.208999816 2023-11-01T04:16:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.373 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:39.374 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.421896+0000", 2023-11-01T04:16:39.374 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000221282 2023-11-01T04:16:39.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.374 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:39.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.423518+0000", 2023-11-01T04:16:39.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001622545 2023-11-01T04:16:39.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:39.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.426160+0000", 2023-11-01T04:16:39.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026410499999999998 2023-11-01T04:16:39.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:39.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.727590+0000", 2023-11-01T04:16:39.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30143093500000001 2023-11-01T04:16:39.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.727699+0000", 2023-11-01T04:16:39.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010878999999999999 2023-11-01T04:16:39.377 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.377 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.378 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.378 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:413 3.7s0 3:ec8ceea5:::smithi19039307-25:head [setxattr _header (57) in=7b] snapc 12=[12,11,10,f,e,d,b,4] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-11-01T04:16:39.378 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:34.212706+0000", 2023-11-01T04:16:39.379 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.9376464129999995, 2023-11-01T04:16:39.379 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.51538549600000005, 2023-11-01T04:16:39.379 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.379 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:39.379 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:16:39.379 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:16:39.380 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.380 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 413 2023-11-01T04:16:39.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.380 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.380 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212706+0000", 2023-11-01T04:16:39.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212706+0000", 2023-11-01T04:16:39.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212708+0000", 2023-11-01T04:16:39.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.733e-06 2023-11-01T04:16:39.383 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.383 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.383 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.383 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212761+0000", 2023-11-01T04:16:39.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3742000000000001e-05 2023-11-01T04:16:39.383 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.384 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.384 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212785+0000", 2023-11-01T04:16:39.384 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3427000000000001e-05 2023-11-01T04:16:39.384 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212801+0000", 2023-11-01T04:16:39.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5758999999999999e-05 2023-11-01T04:16:39.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.386 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.424813+0000", 2023-11-01T04:16:39.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.212012758 2023-11-01T04:16:39.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:16:39.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.424971+0000", 2023-11-01T04:16:39.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015783299999999999 2023-11-01T04:16:39.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:39.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.425686+0000", 2023-11-01T04:16:39.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00071460299999999996 2023-11-01T04:16:39.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.388 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.388 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:39.388 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.427284+0000", 2023-11-01T04:16:39.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001598152 2023-11-01T04:16:39.389 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.389 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.389 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:16:39.389 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.727935+0000", 2023-11-01T04:16:39.389 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30065153700000002 2023-11-01T04:16:39.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.390 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.390 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.390 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.728091+0000", 2023-11-01T04:16:39.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000155952 2023-11-01T04:16:39.390 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.391 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.391 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.391 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 54/31 ECSubWriteReply(tid=53, last_complete=54'27, committed=1, applied=1))", 2023-11-01T04:16:39.391 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:34.215278+0000", 2023-11-01T04:16:39.392 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.9350747669999997, 2023-11-01T04:16:39.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201681376, 2023-11-01T04:16:39.392 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:39.392 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:39.392 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:39.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.393 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:39.393 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.215278+0000", 2023-11-01T04:16:39.393 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.393 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:39.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.215278+0000", 2023-11-01T04:16:39.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:39.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:39.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.215279+0000", 2023-11-01T04:16:39.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.733e-06 2023-11-01T04:16:39.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:39.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.215290+0000", 2023-11-01T04:16:39.396 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0433e-05 2023-11-01T04:16:39.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.396 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:39.396 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.215317+0000", 2023-11-01T04:16:39.396 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7033e-05 2023-11-01T04:16:39.397 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.397 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.397 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.397 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.215330+0000", 2023-11-01T04:16:39.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.327e-05 2023-11-01T04:16:39.397 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:39.398 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.416888+0000", 2023-11-01T04:16:39.398 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201557982 2023-11-01T04:16:39.398 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:39.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:39.399 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:39.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.416959+0000", 2023-11-01T04:16:39.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0925000000000003e-05 2023-11-01T04:16:39.399 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.399 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.399 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.400 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:39.400 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:39.400 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:39.400 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-11-01T04:16:39.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-11-01T04:16:39.402 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:39.400+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:39.425 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-01T04:16:39.502 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:39.500+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:39.535 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:39.535 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2023-11-01T04:16:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:444 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~770048,checksum in=4b,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2023-11-01T04:16:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.143530+0000", 2023-11-01T04:16:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.39256632800000002, 2023-11-01T04:16:39.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.39261855899999998, 2023-11-01T04:16:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.537 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 444 2023-11-01T04:16:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.143530+0000", 2023-11-01T04:16:39.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.143530+0000", 2023-11-01T04:16:39.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.143534+0000", 2023-11-01T04:16:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2940000000000001e-06 2023-11-01T04:16:39.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.143566+0000", 2023-11-01T04:16:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2098999999999999e-05 2023-11-01T04:16:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.143602+0000", 2023-11-01T04:16:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6301999999999999e-05 2023-11-01T04:16:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243774+0000", 2023-11-01T04:16:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100171959 2023-11-01T04:16:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344071+0000", 2023-11-01T04:16:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100297211 2023-11-01T04:16:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2023-11-01T04:16:39.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344083+0000", 2023-11-01T04:16:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.203e-05 2023-11-01T04:16:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344705+0000", 2023-11-01T04:16:39.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000621515 2023-11-01T04:16:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344857+0000", 2023-11-01T04:16:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000152008 2023-11-01T04:16:39.546 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.547 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:445 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~770048,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2023-11-01T04:16:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.243815+0000", 2023-11-01T04:16:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.292281976, 2023-11-01T04:16:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29248819500000001, 2023-11-01T04:16:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:39.548 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 445 2023-11-01T04:16:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.549 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243815+0000", 2023-11-01T04:16:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243815+0000", 2023-11-01T04:16:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243816+0000", 2023-11-01T04:16:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5999999999999999e-06 2023-11-01T04:16:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243826+0000", 2023-11-01T04:16:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4029999999999996e-06 2023-11-01T04:16:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.553 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243844+0000", 2023-11-01T04:16:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8076000000000001e-05 2023-11-01T04:16:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243855+0000", 2023-11-01T04:16:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.17e-05 2023-11-01T04:16:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344243+0000", 2023-11-01T04:16:39.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100387278 2023-11-01T04:16:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344254+0000", 2023-11-01T04:16:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1e-05 2023-11-01T04:16:39.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.347410+0000", 2023-11-01T04:16:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031561829999999999 2023-11-01T04:16:39.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.347553+0000", 2023-11-01T04:16:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000142779 2023-11-01T04:16:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.558 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:446 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~770048,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2023-11-01T04:16:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.243920+0000", 2023-11-01T04:16:39.559 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.29217679299999999, 2023-11-01T04:16:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.292504388, 2023-11-01T04:16:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:39.560 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 446 2023-11-01T04:16:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:39.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243920+0000", 2023-11-01T04:16:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:39.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243920+0000", 2023-11-01T04:16:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243921+0000", 2023-11-01T04:16:39.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3399999999999998e-07 2023-11-01T04:16:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243926+0000", 2023-11-01T04:16:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.096e-06 2023-11-01T04:16:39.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243941+0000", 2023-11-01T04:16:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5569e-05 2023-11-01T04:16:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243951+0000", 2023-11-01T04:16:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4499999999999993e-06 2023-11-01T04:16:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448384+0000", 2023-11-01T04:16:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20443328799999999 2023-11-01T04:16:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.567 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448507+0000", 2023-11-01T04:16:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012274000000000001 2023-11-01T04:16:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 59/31 ECSubReadReply(tid=26, attrs_read=0))", 2023-11-01T04:16:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.446742+0000", 2023-11-01T04:16:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.089354792000000002, 2023-11-01T04:16:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.089777830000000003, 2023-11-01T04:16:39.569 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.446742+0000", 2023-11-01T04:16:39.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.446742+0000", 2023-11-01T04:16:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.446742+0000", 2023-11-01T04:16:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0399999999999996e-07 2023-11-01T04:16:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.446929+0000", 2023-11-01T04:16:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000186761 2023-11-01T04:16:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.446941+0000", 2023-11-01T04:16:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.204e-05 2023-11-01T04:16:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.446947+0000", 2023-11-01T04:16:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.144e-06 2023-11-01T04:16:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 59/31 ECSubReadReply(tid=25, attrs_read=0))", 2023-11-01T04:16:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.448345+0000", 2023-11-01T04:16:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.087751787999999997, 2023-11-01T04:16:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.088245193999999999, 2023-11-01T04:16:39.576 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448345+0000", 2023-11-01T04:16:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448345+0000", 2023-11-01T04:16:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448345+0000", 2023-11-01T04:16:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448345+0000", 2023-11-01T04:16:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448401+0000", 2023-11-01T04:16:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6011000000000003e-05 2023-11-01T04:16:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448409+0000", 2023-11-01T04:16:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4339999999999992e-06 2023-11-01T04:16:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:39.582 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpRead(3.0s0 59/31 ECSubRead(tid=27, to_read={3:0e42a959:::smithi19039307-46:head=0,192512,0}, subchunks={3:0e42a959:::smithi19039307-46:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.448708+0000", 2023-11-01T04:16:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.087388789999999994, 2023-11-01T04:16:39.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.087957252999999999, 2023-11-01T04:16:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:39.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448708+0000", 2023-11-01T04:16:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448708+0000", 2023-11-01T04:16:39.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448708+0000", 2023-11-01T04:16:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448708+0000", 2023-11-01T04:16:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448786+0000", 2023-11-01T04:16:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8313999999999996e-05 2023-11-01T04:16:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448817+0000", 2023-11-01T04:16:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0833000000000001e-05 2023-11-01T04:16:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 59/31 ECSubReadReply(tid=26, attrs_read=0))", 2023-11-01T04:16:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.449045+0000", 2023-11-01T04:16:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.087051740000000002, 2023-11-01T04:16:39.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.087685854999999993, 2023-11-01T04:16:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:39.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449045+0000", 2023-11-01T04:16:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449045+0000", 2023-11-01T04:16:39.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449045+0000", 2023-11-01T04:16:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449045+0000", 2023-11-01T04:16:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449065+0000", 2023-11-01T04:16:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9661999999999999e-05 2023-11-01T04:16:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449074+0000", 2023-11-01T04:16:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2739999999999996e-06 2023-11-01T04:16:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:39.596 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 59/31 ECSubReadReply(tid=27, attrs_read=0))", 2023-11-01T04:16:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.449306+0000", 2023-11-01T04:16:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.086790767000000005, 2023-11-01T04:16:39.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.087489955999999994, 2023-11-01T04:16:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:39.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449306+0000", 2023-11-01T04:16:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449306+0000", 2023-11-01T04:16:39.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449307+0000", 2023-11-01T04:16:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7000000000000003e-07 2023-11-01T04:16:39.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449563+0000", 2023-11-01T04:16:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025623699999999998 2023-11-01T04:16:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449586+0000", 2023-11-01T04:16:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2436999999999999e-05 2023-11-01T04:16:39.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449596+0000", 2023-11-01T04:16:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0642000000000001e-05 2023-11-01T04:16:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.603 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 59/31 ECSubReadReply(tid=27, attrs_read=0))", 2023-11-01T04:16:39.604 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.449428+0000", 2023-11-01T04:16:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.086668994999999999, 2023-11-01T04:16:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.087436445000000002, 2023-11-01T04:16:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:39.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449428+0000", 2023-11-01T04:16:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449428+0000", 2023-11-01T04:16:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:39.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449428+0000", 2023-11-01T04:16:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.16e-07 2023-11-01T04:16:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449666+0000", 2023-11-01T04:16:39.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023703099999999999 2023-11-01T04:16:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449678+0000", 2023-11-01T04:16:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2687999999999999e-05 2023-11-01T04:16:39.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449688+0000", 2023-11-01T04:16:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.482e-06 2023-11-01T04:16:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.610 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 59/31 ECSubReadReply(tid=26, attrs_read=0))", 2023-11-01T04:16:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.451250+0000", 2023-11-01T04:16:39.611 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.084847043999999996, 2023-11-01T04:16:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.085681372000000006, 2023-11-01T04:16:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451250+0000", 2023-11-01T04:16:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:39.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451250+0000", 2023-11-01T04:16:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:39.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451251+0000", 2023-11-01T04:16:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.057e-06 2023-11-01T04:16:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451459+0000", 2023-11-01T04:16:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020849700000000001 2023-11-01T04:16:39.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451476+0000", 2023-11-01T04:16:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6490999999999998e-05 2023-11-01T04:16:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451487+0000", 2023-11-01T04:16:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1094000000000001e-05 2023-11-01T04:16:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.617 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 59/31 ECSubReadReply(tid=27, attrs_read=0))", 2023-11-01T04:16:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.451503+0000", 2023-11-01T04:16:39.618 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.084593974000000002, 2023-11-01T04:16:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.085494688999999999, 2023-11-01T04:16:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451503+0000", 2023-11-01T04:16:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451503+0000", 2023-11-01T04:16:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:39.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451503+0000", 2023-11-01T04:16:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0400000000000002e-07 2023-11-01T04:16:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451679+0000", 2023-11-01T04:16:39.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017560000000000001 2023-11-01T04:16:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451711+0000", 2023-11-01T04:16:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2140000000000001e-05 2023-11-01T04:16:39.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451721+0000", 2023-11-01T04:16:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0052e-05 2023-11-01T04:16:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.624 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2023-11-01T04:16:39.625 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 11 2023-11-01T04:16:39.625 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:39.626 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:39.627 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-11-01T04:16:39.627 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:39.628 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-01T04:16:39.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-11-01T04:16:39.628 INFO:tasks.rados.rados.0.smithi190.stdout:180: expect (ObjNum 6 snap 0 seq_num 6) 2023-11-01T04:16:39.629 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:39.625+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:39.631 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:39.632 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-01T04:16:39.651 INFO:tasks.rados.rados.0.smithi190.stdout:179: expect (ObjNum 45 snap 0 seq_num 45) 2023-11-01T04:16:39.652 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:39.652 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-01T04:16:39.668 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:39.668 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-01T04:16:39.688 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:39.689 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-01T04:16:39.709 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:39.710 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-01T04:16:39.727 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:39.722+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:39.730 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:39.730 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-01T04:16:39.746 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:39.746 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-01T04:16:39.755 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:39.756 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:39.767 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:39.767 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-01T04:16:39.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-11-01T04:16:39.787 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:39.787 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-01T04:16:39.807 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:39.807 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-01T04:16:39.823 INFO:teuthology.orchestra.run.smithi153.stdout:osd.12: {} 2023-11-01T04:16:39.823 INFO:teuthology.orchestra.run.smithi153.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-01T04:16:39.827 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:39.828+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:39.843 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:39.843 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-01T04:16:39.864 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:39.864 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-01T04:16:39.881 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:39.882 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-01T04:16:39.929 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:39.929+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:39.933 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:39.933 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:39.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-01T04:16:40.031 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:40.029+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:40.068 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:40.069 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:40.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-01T04:16:40.132 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:40.131+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:40.153 INFO:teuthology.orchestra.run.smithi153.stderr:nodeep-scrub is unset 2023-11-01T04:16:40.161 INFO:tasks.rados.rados.0.smithi190.stdout:179: done (2 left) 2023-11-01T04:16:40.161 INFO:tasks.rados.rados.0.smithi190.stdout:180: done (1 left) 2023-11-01T04:16:40.161 INFO:tasks.rados.rados.0.smithi190.stdout:181: done (0 left) 2023-11-01T04:16:40.162 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 13 current snap is 16 2023-11-01T04:16:40.162 INFO:tasks.rados.rados.0.smithi190.stdout:182: seq_num 82 ranges {212992=622592,835584=475136,1310720=294912} 2023-11-01T04:16:40.167 INFO:tasks.rados.rados.0.smithi190.stdout:182: writing smithi19039307-13 from 212992 to 835584 tid 1 2023-11-01T04:16:40.171 INFO:tasks.rados.rados.0.smithi190.stdout:182: writing smithi19039307-13 from 835584 to 1310720 tid 2 2023-11-01T04:16:40.174 INFO:tasks.rados.rados.0.smithi190.stdout:182: writing smithi19039307-13 from 1310720 to 1605632 tid 3 2023-11-01T04:16:40.174 INFO:tasks.rados.rados.0.smithi190.stdout:183: copy_from oid 15 from oid 5 current snap is 16 2023-11-01T04:16:40.175 INFO:tasks.rados.rados.0.smithi190.stdout:184: rollback oid 19 current snap is 16 2023-11-01T04:16:40.175 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 19 to 15 2023-11-01T04:16:40.175 INFO:tasks.rados.rados.0.smithi190.stdout:185: copy_from oid 21 from oid 24 current snap is 16 2023-11-01T04:16:40.175 INFO:tasks.rados.rados.0.smithi190.stdout:186: copy_from oid 3 from oid 7 current snap is 16 2023-11-01T04:16:40.175 INFO:tasks.rados.rados.0.smithi190.stdout:187: read oid 33 snap -1 2023-11-01T04:16:40.176 INFO:tasks.rados.rados.0.smithi190.stdout:187: expect (ObjNum 15 snap 0 seq_num 15) 2023-11-01T04:16:40.176 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 16 current snap is 16 2023-11-01T04:16:40.176 INFO:tasks.rados.rados.0.smithi190.stdout:188: seq_num 83 ranges {1622016=638976,2260992=622592,2883584=147456} 2023-11-01T04:16:40.192 INFO:tasks.rados.rados.0.smithi190.stdout:188: writing smithi19039307-16 from 1622016 to 2260992 tid 1 2023-11-01T04:16:40.197 INFO:tasks.rados.rados.0.smithi190.stdout:188: writing smithi19039307-16 from 2260992 to 2883584 tid 2 2023-11-01T04:16:40.198 INFO:tasks.rados.rados.0.smithi190.stdout:188: writing smithi19039307-16 from 2883584 to 3031040 tid 3 2023-11-01T04:16:40.198 INFO:tasks.rados.rados.0.smithi190.stdout:189: rmattr oid 34 current snap is 16 2023-11-01T04:16:40.198 INFO:tasks.rados.rados.0.smithi190.stdout:190: snap_remove snap 10 2023-11-01T04:16:40.209 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:40.210 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:40.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-01T04:16:40.232 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:40.229+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:40.333 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:40.330+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:40.350 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:40.350 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:40.364 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-01T04:16:40.451 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:40.450+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:40.534 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:40.534 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:40.535 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:40.536+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:40.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-01T04:16:40.636 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:40.633+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:40.710 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:40.710 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:40.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-01T04:16:40.772 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:16:40.770+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:16:40.837 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:40.834+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:40.848 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:40.848 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:40.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-01T04:16:40.948 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:40.946+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:41.027 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:41.027 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:41.039 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:41.035+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:41.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-01T04:16:41.129 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2023-11-01T04:16:41.129 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-01T04:16:41.130 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2023-11-01T04:16:41.130 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-11-01T04:16:41.130 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-11-01T04:16:41.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok config set filestore_inject_stall 3 2023-11-01T04:16:41.164 INFO:tasks.rados.rados.0.smithi190.stdout:190: done (8 left) 2023-11-01T04:16:41.164 INFO:tasks.rados.rados.0.smithi190.stdout:191: read oid 25 snap -1 2023-11-01T04:16:41.165 INFO:tasks.rados.rados.0.smithi190.stdout:191: expect (ObjNum 76 snap 13 seq_num 76) 2023-11-01T04:16:41.165 INFO:tasks.rados.rados.0.smithi190.stdout:192: rmattr oid 1 current snap is 16 2023-11-01T04:16:41.165 INFO:tasks.rados.rados.0.smithi190.stdout:192: done (9 left) 2023-11-01T04:16:41.165 INFO:tasks.rados.rados.0.smithi190.stdout:193: snap_create 2023-11-01T04:16:41.165 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2023-11-01T04:16:41.166 INFO:tasks.rados.rados.0.smithi190.stdout:182: finishing write tid 1 to smithi19039307-13 2023-11-01T04:16:41.166 INFO:tasks.rados.rados.0.smithi190.stdout:182: finishing write tid 2 to smithi19039307-13 2023-11-01T04:16:41.166 INFO:tasks.rados.rados.0.smithi190.stdout:185: finishing copy_from racing read to smithi19039307-21 2023-11-01T04:16:41.166 INFO:tasks.rados.rados.0.smithi190.stdout:184: finishing rollback tid 0 to smithi19039307-19 2023-11-01T04:16:41.166 INFO:tasks.rados.rados.0.smithi190.stdout:184: finishing rollback tid 1 to smithi19039307-19 2023-11-01T04:16:41.166 INFO:tasks.rados.rados.0.smithi190.stdout:182: finishing write tid 3 to smithi19039307-13 2023-11-01T04:16:41.167 INFO:tasks.rados.rados.0.smithi190.stdout:184: finishing rollback tid 2 to smithi19039307-19 2023-11-01T04:16:41.167 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 19 v 23 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-01T04:16:41.167 INFO:tasks.rados.rados.0.smithi190.stdout:183: finishing copy_from racing read to smithi19039307-15 2023-11-01T04:16:41.167 INFO:tasks.rados.rados.0.smithi190.stdout:188: finishing write tid 1 to smithi19039307-16 2023-11-01T04:16:41.167 INFO:tasks.rados.rados.0.smithi190.stdout:182: finishing write tid 4 to smithi19039307-13 2023-11-01T04:16:41.167 INFO:tasks.rados.rados.0.smithi190.stdout:187: expect (ObjNum 15 snap 0 seq_num 15) 2023-11-01T04:16:41.168 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:41.162+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:41.177 INFO:tasks.rados.rados.0.smithi190.stdout:185: finishing copy_from to smithi19039307-21 2023-11-01T04:16:41.177 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 21 v 29 (ObjNum 59 snap 5 seq_num 59) dirty exists 2023-11-01T04:16:41.177 INFO:tasks.rados.rados.0.smithi190.stdout:186: finishing copy_from racing read to smithi19039307-3 2023-11-01T04:16:41.177 INFO:tasks.rados.rados.0.smithi190.stdout:183: finishing copy_from to smithi19039307-15 2023-11-01T04:16:41.178 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 15 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2023-11-01T04:16:41.178 INFO:tasks.rados.rados.0.smithi190.stdout:188: finishing write tid 2 to smithi19039307-16 2023-11-01T04:16:41.178 INFO:tasks.rados.rados.0.smithi190.stdout:188: finishing write tid 3 to smithi19039307-16 2023-11-01T04:16:41.178 INFO:tasks.rados.rados.0.smithi190.stdout:188: finishing write tid 4 to smithi19039307-16 2023-11-01T04:16:41.178 INFO:tasks.rados.rados.0.smithi190.stdout:182: finishing write tid 5 to smithi19039307-13 2023-11-01T04:16:41.179 INFO:tasks.rados.rados.0.smithi190.stdout:182: oid 13 updating version 0 to 48 2023-11-01T04:16:41.179 INFO:tasks.rados.rados.0.smithi190.stdout:182: oid 13 updating version 48 to 50 2023-11-01T04:16:41.179 INFO:tasks.rados.rados.0.smithi190.stdout:182: oid 13 version 50 is already newer than 49 2023-11-01T04:16:41.179 INFO:tasks.rados.rados.0.smithi190.stdout:182: oid 13 version 50 is already newer than 47 2023-11-01T04:16:41.179 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 13 v 50 (ObjNum 82 snap 16 seq_num 82) dirty exists 2023-11-01T04:16:41.179 INFO:tasks.rados.rados.0.smithi190.stdout:182: left oid 13 (ObjNum 82 snap 16 seq_num 82) 2023-11-01T04:16:41.180 INFO:tasks.rados.rados.0.smithi190.stdout:188: finishing write tid 5 to smithi19039307-16 2023-11-01T04:16:41.180 INFO:tasks.rados.rados.0.smithi190.stdout:188: oid 16 updating version 0 to 53 2023-11-01T04:16:41.180 INFO:tasks.rados.rados.0.smithi190.stdout:188: oid 16 updating version 53 to 56 2023-11-01T04:16:41.180 INFO:tasks.rados.rados.0.smithi190.stdout:188: oid 16 version 56 is already newer than 55 2023-11-01T04:16:41.180 INFO:tasks.rados.rados.0.smithi190.stdout:188: oid 16 version 56 is already newer than 54 2023-11-01T04:16:41.180 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 16 v 56 (ObjNum 83 snap 16 seq_num 83) dirty exists 2023-11-01T04:16:41.181 INFO:tasks.rados.rados.0.smithi190.stdout:188: left oid 16 (ObjNum 83 snap 16 seq_num 83) 2023-11-01T04:16:41.181 INFO:tasks.rados.rados.0.smithi190.stdout:186: finishing copy_from to smithi19039307-3 2023-11-01T04:16:41.181 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 3 v 14 (ObjNum 6 snap 0 seq_num 6) dirty exists 2023-11-01T04:16:41.208 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:41.209 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:41.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-01T04:16:41.241 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:41.240+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:41.257 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:41.257 INFO:teuthology.orchestra.run.smithi190.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-11-01T04:16:41.257 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:41.342 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:41.338+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:41.377 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:41.377 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:41.392 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-01T04:16:41.443 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:41.440+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:41.498 INFO:tasks.rados.rados.0.smithi190.stdout:191: expect (ObjNum 76 snap 13 seq_num 76) 2023-11-01T04:16:41.527 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:41.527 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:41.541 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-01T04:16:41.543 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:41.539+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:41.643 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:41.642+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:41.703 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:41.703 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:41.717 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-01T04:16:41.744 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:41.743+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:41.839 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:41.839 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:41.845 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:41.843+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:41.853 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-01T04:16:41.900 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-01T04:16:41.946 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:41.945+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:41.972 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:41.972 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:41.986 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-01T04:16:42.045 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:42.045 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-11-01T04:16:42.047 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:42.043+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:42.066 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-01T04:16:42.086 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:42.087 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-01T04:16:42.106 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:42.106 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:42.108 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:42.108 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-01T04:16:42.120 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-01T04:16:42.128 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:42.128 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-01T04:16:42.147 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:42.146+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:42.149 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:42.149 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-01T04:16:42.169 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:42.170 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-01T04:16:42.177 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-11-01T04:16:42.187 INFO:tasks.rados.rados.0.smithi190.stdout:182: done (9 left) 2023-11-01T04:16:42.187 INFO:tasks.rados.rados.0.smithi190.stdout:183: done (8 left) 2023-11-01T04:16:42.187 INFO:tasks.rados.rados.0.smithi190.stdout:184: done (7 left) 2023-11-01T04:16:42.188 INFO:tasks.rados.rados.0.smithi190.stdout:185: done (6 left) 2023-11-01T04:16:42.188 INFO:tasks.rados.rados.0.smithi190.stdout:186: done (5 left) 2023-11-01T04:16:42.188 INFO:tasks.rados.rados.0.smithi190.stdout:187: done (4 left) 2023-11-01T04:16:42.188 INFO:tasks.rados.rados.0.smithi190.stdout:188: done (3 left) 2023-11-01T04:16:42.188 INFO:tasks.rados.rados.0.smithi190.stdout:189: done (2 left) 2023-11-01T04:16:42.188 INFO:tasks.rados.rados.0.smithi190.stdout:191: done (1 left) 2023-11-01T04:16:42.189 INFO:tasks.rados.rados.0.smithi190.stdout:193: done (0 left) 2023-11-01T04:16:42.189 INFO:tasks.rados.rados.0.smithi190.stdout:194: copy_from oid 17 from oid 8 current snap is 17 2023-11-01T04:16:42.189 INFO:tasks.rados.rados.0.smithi190.stdout:195: read oid 32 snap -1 2023-11-01T04:16:42.189 INFO:tasks.rados.rados.0.smithi190.stdout:195: expect (ObjNum 53 snap 1 seq_num 53) 2023-11-01T04:16:42.189 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 43 current snap is 17 2023-11-01T04:16:42.189 INFO:tasks.rados.rados.0.smithi190.stdout:196: seq_num 84 ranges {1245184=655360,1900544=638976,2539520=131072} 2023-11-01T04:16:42.195 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:42.195 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-01T04:16:42.205 INFO:tasks.rados.rados.0.smithi190.stdout:196: writing smithi19039307-43 from 1245184 to 1900544 tid 1 2023-11-01T04:16:42.212 INFO:tasks.rados.rados.0.smithi190.stdout:196: writing smithi19039307-43 from 1900544 to 2539520 tid 2 2023-11-01T04:16:42.213 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:42.214 INFO:tasks.rados.rados.0.smithi190.stdout:196: writing smithi19039307-43 from 2539520 to 2670592 tid 3 2023-11-01T04:16:42.214 INFO:tasks.rados.rados.0.smithi190.stdout:197: setattr oid 39 current snap is 17 2023-11-01T04:16:42.214 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-01T04:16:42.215 INFO:tasks.rados.rados.0.smithi190.stdout:198: read oid 14 snap 13 2023-11-01T04:16:42.216 INFO:tasks.rados.rados.0.smithi190.stdout:198: expect deleted 2023-11-01T04:16:42.216 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 16 current snap is 17 2023-11-01T04:16:42.216 INFO:tasks.rados.rados.0.smithi190.stdout:199: seq_num 85 ranges {3031040=671744,3702784=770048} 2023-11-01T04:16:42.234 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:42.234 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-01T04:16:42.244 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:42.245 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2023-11-01T04:16:42.245 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2023-11-01T04:16:42.245 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:16:42.245 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.245 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.es3 36/31 ECSubWrite(tid=12, reqid=client.4559.0:133, at_version=36'7, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:42.246 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.169895+0000", 2023-11-01T04:16:42.246 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 27.073990488, 2023-11-01T04:16:42.246 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103016208, 2023-11-01T04:16:42.246 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.246 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.246 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.247 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.247 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.247 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.169895+0000", 2023-11-01T04:16:42.247 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.248 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.248 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.248 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.169895+0000", 2023-11-01T04:16:42.248 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.248 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.249 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.249 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.169897+0000", 2023-11-01T04:16:42.249 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.466e-06 2023-11-01T04:16:42.249 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.249 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.250 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.250 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.170419+0000", 2023-11-01T04:16:42.250 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00052250599999999995 2023-11-01T04:16:42.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.250 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.250 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.251 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.170457+0000", 2023-11-01T04:16:42.251 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.7945999999999999e-05 2023-11-01T04:16:42.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.251 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.251 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.170474+0000", 2023-11-01T04:16:42.252 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7045999999999999e-05 2023-11-01T04:16:42.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.252 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.252 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.252 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.270728+0000", 2023-11-01T04:16:42.252 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100254038 2023-11-01T04:16:42.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.253 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.253 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.270744+0000", 2023-11-01T04:16:42.253 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.507e-05 2023-11-01T04:16:42.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.254 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.254 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.272841+0000", 2023-11-01T04:16:42.254 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0020970310000000001 2023-11-01T04:16:42.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.255 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.255 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.272912+0000", 2023-11-01T04:16:42.255 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1105000000000004e-05 2023-11-01T04:16:42.255 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.255 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.256 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.256 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.256 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.as1 36/31 ECSubWrite(tid=24, reqid=client.4559.0:140, at_version=36'4, trim_to=0'0, roll_forward_to=34'2))", 2023-11-01T04:16:42.256 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.199076+0000", 2023-11-01T04:16:42.256 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 27.044809830999998, 2023-11-01T04:16:42.257 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20133796800000001, 2023-11-01T04:16:42.257 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.257 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.257 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.257 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.257 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.258 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199076+0000", 2023-11-01T04:16:42.258 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.258 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.258 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.259 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199076+0000", 2023-11-01T04:16:42.259 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.259 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.259 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.259 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.259 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199078+0000", 2023-11-01T04:16:42.260 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.623e-06 2023-11-01T04:16:42.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.260 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.260 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.260 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199121+0000", 2023-11-01T04:16:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3727999999999999e-05 2023-11-01T04:16:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.261 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199278+0000", 2023-11-01T04:16:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015602899999999999 2023-11-01T04:16:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299400+0000", 2023-11-01T04:16:42.262 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100122192 2023-11-01T04:16:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399577+0000", 2023-11-01T04:16:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10017696500000001 2023-11-01T04:16:42.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399597+0000", 2023-11-01T04:16:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0213000000000001e-05 2023-11-01T04:16:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.264 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.400376+0000", 2023-11-01T04:16:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00077926799999999995 2023-11-01T04:16:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.265 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.400414+0000", 2023-11-01T04:16:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.7950000000000001e-05 2023-11-01T04:16:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.0s2 36/31 ECSubWrite(tid=6, reqid=client.4559.0:137, at_version=0'0, trim_to=0'0, roll_forward_to=36'4))", 2023-11-01T04:16:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.609108+0000", 2023-11-01T04:16:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 26.634777841999998, 2023-11-01T04:16:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20084221199999999, 2023-11-01T04:16:42.267 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.609108+0000", 2023-11-01T04:16:42.268 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.609108+0000", 2023-11-01T04:16:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.609109+0000", 2023-11-01T04:16:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.257e-06 2023-11-01T04:16:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.270 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.609125+0000", 2023-11-01T04:16:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5305e-05 2023-11-01T04:16:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.271 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.609143+0000", 2023-11-01T04:16:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8746e-05 2023-11-01T04:16:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.272 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.273 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709231+0000", 2023-11-01T04:16:42.273 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100087153 2023-11-01T04:16:42.273 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.273 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.273 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.273 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709326+0000", 2023-11-01T04:16:42.274 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5130000000000006e-05 2023-11-01T04:16:42.274 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.274 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.274 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.274 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.709341+0000", 2023-11-01T04:16:42.274 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5699999999999999e-05 2023-11-01T04:16:42.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.275 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.275 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.275 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.809924+0000", 2023-11-01T04:16:42.275 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10058260099999999 2023-11-01T04:16:42.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.276 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.276 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.276 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.809950+0000", 2023-11-01T04:16:42.276 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6319999999999999e-05 2023-11-01T04:16:42.276 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.276 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.277 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.277 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.277 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.277 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds1 36/31 ECSubWrite(tid=30, reqid=client.4559.0:143, at_version=36'14, trim_to=0'0, roll_forward_to=36'13))", 2023-11-01T04:16:42.277 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.668264+0000", 2023-11-01T04:16:42.277 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 26.575621841, 2023-11-01T04:16:42.278 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20128238700000001, 2023-11-01T04:16:42.278 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.278 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.278 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.278 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.278 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.279 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.668264+0000", 2023-11-01T04:16:42.279 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.279 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.279 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.668264+0000", 2023-11-01T04:16:42.280 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.280 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.280 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.280 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.668265+0000", 2023-11-01T04:16:42.280 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0699999999999996e-07 2023-11-01T04:16:42.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.281 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.281 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.281 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.668318+0000", 2023-11-01T04:16:42.281 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3316000000000002e-05 2023-11-01T04:16:42.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.282 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.282 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.282 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.668335+0000", 2023-11-01T04:16:42.282 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6935999999999998e-05 2023-11-01T04:16:42.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.283 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.283 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.283 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.768415+0000", 2023-11-01T04:16:42.283 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100080154 2023-11-01T04:16:42.283 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.284 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.284 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.284 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.768508+0000", 2023-11-01T04:16:42.284 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2936000000000006e-05 2023-11-01T04:16:42.284 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.284 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.285 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.285 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.768521+0000", 2023-11-01T04:16:42.285 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2683999999999999e-05 2023-11-01T04:16:42.285 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.285 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.285 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.286 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.869474+0000", 2023-11-01T04:16:42.286 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100952606 2023-11-01T04:16:42.286 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.286 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.286 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.286 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.869547+0000", 2023-11-01T04:16:42.287 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2947999999999993e-05 2023-11-01T04:16:42.287 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.287 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.287 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.287 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.288 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.288 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.ds1 36/31 ECSubRead(tid=33, to_read={3:b31c2ca6:::smithi19039307-48:head=0,4096,0}, subchunks={3:b31c2ca6:::smithi19039307-48:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:42.288 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.872270+0000", 2023-11-01T04:16:42.288 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 26.371616298999999, 2023-11-01T04:16:42.288 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.199929154, 2023-11-01T04:16:42.288 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.289 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.289 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.289 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.289 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.289 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.872270+0000", 2023-11-01T04:16:42.289 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.290 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.290 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.290 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.290 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.872270+0000", 2023-11-01T04:16:42.290 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.290 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.291 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.291 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.291 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.872271+0000", 2023-11-01T04:16:42.291 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.517e-06 2023-11-01T04:16:42.291 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.291 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.292 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.292 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.872291+0000", 2023-11-01T04:16:42.292 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0060000000000001e-05 2023-11-01T04:16:42.292 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.292 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.292 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.293 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.872331+0000", 2023-11-01T04:16:42.293 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0234999999999997e-05 2023-11-01T04:16:42.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.293 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.293 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.293 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.872362+0000", 2023-11-01T04:16:42.294 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0315999999999999e-05 2023-11-01T04:16:42.294 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.294 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.294 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.294 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.071857+0000", 2023-11-01T04:16:42.295 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19949471299999999 2023-11-01T04:16:42.295 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.295 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.295 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.295 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.072199+0000", 2023-11-01T04:16:42.295 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00034231300000000001 2023-11-01T04:16:42.296 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.296 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.296 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.296 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.296 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.es3 37/31 ECSubWrite(tid=21, reqid=client.4559.0:186, at_version=37'14, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:16:42.297 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.967923+0000", 2023-11-01T04:16:42.297 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 25.275963199, 2023-11-01T04:16:42.297 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10252109500000001, 2023-11-01T04:16:42.297 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.297 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.297 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.298 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.298 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.298 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967923+0000", 2023-11-01T04:16:42.298 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.298 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.298 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.299 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.299 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967923+0000", 2023-11-01T04:16:42.299 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.299 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.299 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.300 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.300 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967924+0000", 2023-11-01T04:16:42.300 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3600000000000002e-07 2023-11-01T04:16:42.300 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.300 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.300 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.301 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967943+0000", 2023-11-01T04:16:42.301 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9168999999999999e-05 2023-11-01T04:16:42.301 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.301 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.301 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.301 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967958+0000", 2023-11-01T04:16:42.302 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4802e-05 2023-11-01T04:16:42.302 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.302 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.302 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.302 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967965+0000", 2023-11-01T04:16:42.302 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.7130000000000005e-06 2023-11-01T04:16:42.303 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.303 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.303 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.303 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.065569+0000", 2023-11-01T04:16:42.303 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.097603500999999995 2023-11-01T04:16:42.303 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.304 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.304 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.304 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.065584+0000", 2023-11-01T04:16:42.304 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5438e-05 2023-11-01T04:16:42.304 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.304 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.305 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.305 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.070416+0000", 2023-11-01T04:16:42.305 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0048317789999999996 2023-11-01T04:16:42.305 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.305 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.305 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.306 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.070444+0000", 2023-11-01T04:16:42.306 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7857000000000001e-05 2023-11-01T04:16:42.306 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.306 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.306 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.306 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.307 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.307 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds1 37/31 ECSubWrite(tid=44, reqid=client.4559.0:189, at_version=37'19, trim_to=0'0, roll_forward_to=36'17))", 2023-11-01T04:16:42.307 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:17.267187+0000", 2023-11-01T04:16:42.307 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 24.976698606999999, 2023-11-01T04:16:42.307 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.102825324, 2023-11-01T04:16:42.308 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.308 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.308 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.308 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.308 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.309 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267187+0000", 2023-11-01T04:16:42.309 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.309 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.309 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.309 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.309 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267187+0000", 2023-11-01T04:16:42.310 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.310 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.310 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.310 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267189+0000", 2023-11-01T04:16:42.310 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.807e-06 2023-11-01T04:16:42.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.311 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.311 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.311 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267255+0000", 2023-11-01T04:16:42.311 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6070999999999998e-05 2023-11-01T04:16:42.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.312 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.312 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.312 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267294+0000", 2023-11-01T04:16:42.312 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8716e-05 2023-11-01T04:16:42.312 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.312 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.313 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.313 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267318+0000", 2023-11-01T04:16:42.313 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3598999999999999e-05 2023-11-01T04:16:42.313 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.313 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.313 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.314 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.367569+0000", 2023-11-01T04:16:42.314 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100251039 2023-11-01T04:16:42.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.314 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.314 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.314 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.367583+0000", 2023-11-01T04:16:42.315 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4473e-05 2023-11-01T04:16:42.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.315 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.315 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.315 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.369922+0000", 2023-11-01T04:16:42.315 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0023389890000000001 2023-11-01T04:16:42.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.316 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.316 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.316 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.370013+0000", 2023-11-01T04:16:42.316 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0630000000000005e-05 2023-11-01T04:16:42.316 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.317 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.317 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.317 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.317 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.es3 37/31 ECSubWrite(tid=24, reqid=client.4559.0:194, at_version=37'17, trim_to=0'0, roll_forward_to=37'15))", 2023-11-01T04:16:42.317 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:17.480035+0000", 2023-11-01T04:16:42.318 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 24.763851282000001, 2023-11-01T04:16:42.318 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10411511599999999, 2023-11-01T04:16:42.318 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.318 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.318 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.319 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.319 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.319 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.480035+0000", 2023-11-01T04:16:42.319 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.319 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.320 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.320 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.480035+0000", 2023-11-01T04:16:42.320 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.320 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.320 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.321 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.480037+0000", 2023-11-01T04:16:42.321 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9300000000000002e-06 2023-11-01T04:16:42.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.321 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.321 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.321 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.480565+0000", 2023-11-01T04:16:42.322 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00052879800000000003 2023-11-01T04:16:42.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.322 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.322 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.322 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.480604+0000", 2023-11-01T04:16:42.322 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8325000000000003e-05 2023-11-01T04:16:42.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.323 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.323 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.323 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.480626+0000", 2023-11-01T04:16:42.323 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2646e-05 2023-11-01T04:16:42.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.324 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.324 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.324 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.580869+0000", 2023-11-01T04:16:42.324 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100242556 2023-11-01T04:16:42.324 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.324 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.325 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.580883+0000", 2023-11-01T04:16:42.325 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4273e-05 2023-11-01T04:16:42.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.325 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.325 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.326 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.584055+0000", 2023-11-01T04:16:42.326 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0031720569999999998 2023-11-01T04:16:42.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.326 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.326 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.326 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.584150+0000", 2023-11-01T04:16:42.327 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4530999999999993e-05 2023-11-01T04:16:42.327 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.327 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.327 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.327 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.328 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds1 41/31 ECSubWrite(tid=55, reqid=client.4559.0:266, at_version=41'21, trim_to=0'0, roll_forward_to=37'19))", 2023-11-01T04:16:42.328 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.123059+0000", 2023-11-01T04:16:42.328 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 21.120827231, 2023-11-01T04:16:42.328 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103087974, 2023-11-01T04:16:42.328 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.328 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.329 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.329 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.329 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.329 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123059+0000", 2023-11-01T04:16:42.329 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.330 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.330 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.330 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123059+0000", 2023-11-01T04:16:42.330 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.331 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.331 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123061+0000", 2023-11-01T04:16:42.331 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8870000000000001e-06 2023-11-01T04:16:42.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.332 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.332 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123341+0000", 2023-11-01T04:16:42.332 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00028086000000000002 2023-11-01T04:16:42.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.332 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.332 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.333 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123374+0000", 2023-11-01T04:16:42.333 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2852000000000003e-05 2023-11-01T04:16:42.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.333 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.333 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.333 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.223454+0000", 2023-11-01T04:16:42.334 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100079638 2023-11-01T04:16:42.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.334 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.334 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.223532+0000", 2023-11-01T04:16:42.334 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.7715999999999999e-05 2023-11-01T04:16:42.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.335 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.335 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.335 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.223541+0000", 2023-11-01T04:16:42.335 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8550000000000008e-06 2023-11-01T04:16:42.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.336 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.336 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.336 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.226096+0000", 2023-11-01T04:16:42.336 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002555505 2023-11-01T04:16:42.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.336 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.337 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.337 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.226147+0000", 2023-11-01T04:16:42.337 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0661000000000002e-05 2023-11-01T04:16:42.337 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.337 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.337 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.338 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/31 ECSubWrite(tid=236, reqid=client.4559.0:330, at_version=46'36, trim_to=0'0, roll_forward_to=45'33))", 2023-11-01T04:16:42.338 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.282825+0000", 2023-11-01T04:16:42.338 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 15.961061003999999, 2023-11-01T04:16:42.338 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20525125999999999, 2023-11-01T04:16:42.339 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.339 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.339 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.339 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.340 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282825+0000", 2023-11-01T04:16:42.340 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.340 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.340 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282825+0000", 2023-11-01T04:16:42.341 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.341 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.341 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282827+0000", 2023-11-01T04:16:42.341 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2800000000000002e-06 2023-11-01T04:16:42.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.342 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.342 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.342 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282955+0000", 2023-11-01T04:16:42.342 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012810400000000001 2023-11-01T04:16:42.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.343 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.343 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.283001+0000", 2023-11-01T04:16:42.343 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5411999999999998e-05 2023-11-01T04:16:42.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.344 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.344 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383146+0000", 2023-11-01T04:16:42.344 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100144996 2023-11-01T04:16:42.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.344 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.344 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.345 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383332+0000", 2023-11-01T04:16:42.345 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00018672099999999999 2023-11-01T04:16:42.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.345 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.345 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383371+0000", 2023-11-01T04:16:42.346 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8532999999999999e-05 2023-11-01T04:16:42.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.346 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.346 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.346 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.488032+0000", 2023-11-01T04:16:42.346 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104661192 2023-11-01T04:16:42.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.347 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.347 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.488076+0000", 2023-11-01T04:16:42.347 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4022e-05 2023-11-01T04:16:42.347 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.348 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.348 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.348 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.348 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/31 ECSubWrite(tid=237, reqid=client.4559.0:331, at_version=46'37, trim_to=0'0, roll_forward_to=45'33))", 2023-11-01T04:16:42.349 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.383221+0000", 2023-11-01T04:16:42.349 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 15.860665149000001, 2023-11-01T04:16:42.349 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10489668100000001, 2023-11-01T04:16:42.349 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.349 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.349 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.350 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.350 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383221+0000", 2023-11-01T04:16:42.350 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.351 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.351 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383221+0000", 2023-11-01T04:16:42.351 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.352 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.352 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383222+0000", 2023-11-01T04:16:42.352 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.545e-06 2023-11-01T04:16:42.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.352 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.352 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.353 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383256+0000", 2023-11-01T04:16:42.353 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4094999999999998e-05 2023-11-01T04:16:42.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.353 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.353 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.353 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383273+0000", 2023-11-01T04:16:42.354 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6722999999999999e-05 2023-11-01T04:16:42.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.354 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.354 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.354 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383285+0000", 2023-11-01T04:16:42.354 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1728e-05 2023-11-01T04:16:42.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.355 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.355 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.384624+0000", 2023-11-01T04:16:42.355 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001339086 2023-11-01T04:16:42.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.356 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.356 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.356 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.384639+0000", 2023-11-01T04:16:42.356 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5328e-05 2023-11-01T04:16:42.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.357 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.357 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.357 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.488085+0000", 2023-11-01T04:16:42.357 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103445437 2023-11-01T04:16:42.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.357 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.358 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.358 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.488117+0000", 2023-11-01T04:16:42.358 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2738999999999999e-05 2023-11-01T04:16:42.358 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.358 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.359 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.359 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.359 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.2s3 46/31 ECSubRead(tid=238, to_read={3:4a13469b:::smithi19039307-24:head=0,561152,0}, subchunks={3:4a13469b:::smithi19039307-24:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:42.359 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.383307+0000", 2023-11-01T04:16:42.359 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 15.860578719999999, 2023-11-01T04:16:42.360 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104403442, 2023-11-01T04:16:42.360 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.360 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.360 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.360 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.360 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383307+0000", 2023-11-01T04:16:42.361 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.361 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.361 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383307+0000", 2023-11-01T04:16:42.362 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.362 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.362 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383308+0000", 2023-11-01T04:16:42.363 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6699999999999999e-07 2023-11-01T04:16:42.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.363 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.363 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383317+0000", 2023-11-01T04:16:42.363 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9959999999999999e-06 2023-11-01T04:16:42.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.364 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.364 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383331+0000", 2023-11-01T04:16:42.364 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4289e-05 2023-11-01T04:16:42.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.365 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.365 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.365 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383343+0000", 2023-11-01T04:16:42.365 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1986e-05 2023-11-01T04:16:42.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.365 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.366 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.366 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.484996+0000", 2023-11-01T04:16:42.366 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101652828 2023-11-01T04:16:42.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.366 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.367 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.487711+0000", 2023-11-01T04:16:42.367 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0027148760000000002 2023-11-01T04:16:42.367 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.367 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.367 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.368 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.368 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.2s3 46/31 ECSubRead(tid=239, to_read={3:4a13469b:::smithi19039307-24:head=0,561152,0}, subchunks={3:4a13469b:::smithi19039307-24:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:42.368 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.383384+0000", 2023-11-01T04:16:42.368 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 15.860501984999999, 2023-11-01T04:16:42.368 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104738817, 2023-11-01T04:16:42.368 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.369 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.369 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.369 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.369 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.369 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383384+0000", 2023-11-01T04:16:42.369 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.370 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.370 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.370 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383384+0000", 2023-11-01T04:16:42.370 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.371 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.371 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.371 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383384+0000", 2023-11-01T04:16:42.371 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9000000000000002e-07 2023-11-01T04:16:42.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.371 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.372 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383391+0000", 2023-11-01T04:16:42.372 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9870000000000002e-06 2023-11-01T04:16:42.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.372 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.372 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383406+0000", 2023-11-01T04:16:42.373 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4270000000000001e-05 2023-11-01T04:16:42.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.373 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383420+0000", 2023-11-01T04:16:42.374 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3890000000000001e-05 2023-11-01T04:16:42.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.374 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.374 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.374 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.487782+0000", 2023-11-01T04:16:42.374 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10436245399999999 2023-11-01T04:16:42.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.375 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.375 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.488123+0000", 2023-11-01T04:16:42.375 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00034082600000000002 2023-11-01T04:16:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.376 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.2s3 46/31 ECSubRead(tid=240, to_read={3:4a13469b:::smithi19039307-24:head=0,561152,0}, subchunks={3:4a13469b:::smithi19039307-24:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.383437+0000", 2023-11-01T04:16:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 15.860448978999999, 2023-11-01T04:16:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10541188, 2023-11-01T04:16:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.377 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383437+0000", 2023-11-01T04:16:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.378 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383437+0000", 2023-11-01T04:16:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.379 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383437+0000", 2023-11-01T04:16:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5999999999999999e-07 2023-11-01T04:16:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.380 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383445+0000", 2023-11-01T04:16:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6299999999999998e-06 2023-11-01T04:16:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.381 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383457+0000", 2023-11-01T04:16:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1555e-05 2023-11-01T04:16:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.382 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383476+0000", 2023-11-01T04:16:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9397000000000001e-05 2023-11-01T04:16:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.383 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.488221+0000", 2023-11-01T04:16:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104745329 2023-11-01T04:16:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.384 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.488849+0000", 2023-11-01T04:16:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000627609 2023-11-01T04:16:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.385 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.1s1 47/31 ECSubWrite(tid=43, reqid=client.4559.0:355, at_version=47'23, trim_to=0'0, roll_forward_to=46'20))", 2023-11-01T04:16:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.484592+0000", 2023-11-01T04:16:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 14.759293558, 2023-11-01T04:16:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.102474474, 2023-11-01T04:16:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.386 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.387 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.387 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.387 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.387 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.484592+0000", 2023-11-01T04:16:42.388 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.388 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.388 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.388 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.388 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.484592+0000", 2023-11-01T04:16:42.388 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.389 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.389 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.389 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.484594+0000", 2023-11-01T04:16:42.389 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3260000000000001e-06 2023-11-01T04:16:42.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.390 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.390 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.390 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.484755+0000", 2023-11-01T04:16:42.390 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016139 2023-11-01T04:16:42.390 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.390 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.391 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.391 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.484779+0000", 2023-11-01T04:16:42.391 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3419000000000001e-05 2023-11-01T04:16:42.391 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.391 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.391 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.392 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.584938+0000", 2023-11-01T04:16:42.392 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100159436 2023-11-01T04:16:42.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.392 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.392 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.392 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.585078+0000", 2023-11-01T04:16:42.393 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000140414 2023-11-01T04:16:42.393 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.393 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.393 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.393 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.585105+0000", 2023-11-01T04:16:42.394 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6533000000000002e-05 2023-11-01T04:16:42.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.394 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.394 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.394 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.586976+0000", 2023-11-01T04:16:42.394 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001871501 2023-11-01T04:16:42.395 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.395 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.395 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.395 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.587067+0000", 2023-11-01T04:16:42.395 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0455e-05 2023-11-01T04:16:42.395 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.396 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.396 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.396 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.396 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.9s4 53/31 ECSubWrite(tid=49, reqid=client.4559.0:408, at_version=53'39, trim_to=0'0, roll_forward_to=51'37))", 2023-11-01T04:16:42.396 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:33.209188+0000", 2023-11-01T04:16:42.397 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 9.0346980529999996, 2023-11-01T04:16:42.397 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10547720200000001, 2023-11-01T04:16:42.397 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.397 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.397 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.397 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.398 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.398 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.209188+0000", 2023-11-01T04:16:42.398 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.398 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.399 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.399 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.209188+0000", 2023-11-01T04:16:42.399 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.399 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.400 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.400 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.209188+0000", 2023-11-01T04:16:42.400 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9000000000000001e-07 2023-11-01T04:16:42.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.400 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.401 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.209540+0000", 2023-11-01T04:16:42.401 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00035237099999999998 2023-11-01T04:16:42.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.401 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.401 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.401 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.209559+0000", 2023-11-01T04:16:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8059000000000001e-05 2023-11-01T04:16:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.309662+0000", 2023-11-01T04:16:42.402 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100103894 2023-11-01T04:16:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.310907+0000", 2023-11-01T04:16:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0012448769999999999 2023-11-01T04:16:42.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.310919+0000", 2023-11-01T04:16:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1763000000000001e-05 2023-11-01T04:16:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.404 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.314538+0000", 2023-11-01T04:16:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00361854 2023-11-01T04:16:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.405 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.314665+0000", 2023-11-01T04:16:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000127508 2023-11-01T04:16:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.406 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds1 57/31 ECSubWrite(tid=103, reqid=osd.12.0:103, at_version=57'54, trim_to=0'0, roll_forward_to=55'52))", 2023-11-01T04:16:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:37.235875+0000", 2023-11-01T04:16:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.0080109630000003, 2023-11-01T04:16:42.407 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.102507008, 2023-11-01T04:16:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.408 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.235875+0000", 2023-11-01T04:16:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.409 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.235875+0000", 2023-11-01T04:16:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.410 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.235875+0000", 2023-11-01T04:16:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4999999999999999e-07 2023-11-01T04:16:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.235902+0000", 2023-11-01T04:16:42.411 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6475e-05 2023-11-01T04:16:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.235918+0000", 2023-11-01T04:16:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6342999999999999e-05 2023-11-01T04:16:42.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.235927+0000", 2023-11-01T04:16:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0829999999999993e-06 2023-11-01T04:16:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.413 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.414 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.414 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.236013+0000", 2023-11-01T04:16:42.414 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.6005000000000001e-05 2023-11-01T04:16:42.414 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.414 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.415 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.415 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.236021+0000", 2023-11-01T04:16:42.415 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0339999999999997e-06 2023-11-01T04:16:42.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.415 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.415 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.416 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.338293+0000", 2023-11-01T04:16:42.416 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.102272194 2023-11-01T04:16:42.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.416 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.416 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.416 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.338382+0000", 2023-11-01T04:16:42.416 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8623999999999997e-05 2023-11-01T04:16:42.417 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.417 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.417 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.417 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.417 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds1 57/31 ECSubWrite(tid=104, reqid=osd.12.0:104, at_version=57'56, trim_to=0'0, roll_forward_to=55'52))", 2023-11-01T04:16:42.418 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:37.236876+0000", 2023-11-01T04:16:42.418 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.0070102649999999, 2023-11-01T04:16:42.418 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103109016, 2023-11-01T04:16:42.418 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.418 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.419 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.419 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.419 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.419 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.236876+0000", 2023-11-01T04:16:42.419 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.420 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.420 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.420 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.236876+0000", 2023-11-01T04:16:42.420 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.421 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.421 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.421 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.236877+0000", 2023-11-01T04:16:42.421 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.226e-06 2023-11-01T04:16:42.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.421 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.422 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.422 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.236903+0000", 2023-11-01T04:16:42.422 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5965999999999998e-05 2023-11-01T04:16:42.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.422 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.422 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.423 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.236922+0000", 2023-11-01T04:16:42.423 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9533000000000001e-05 2023-11-01T04:16:42.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.423 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.423 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.423 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.236950+0000", 2023-11-01T04:16:42.424 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7594e-05 2023-11-01T04:16:42.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.424 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.424 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.424 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.337287+0000", 2023-11-01T04:16:42.424 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100336699 2023-11-01T04:16:42.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.425 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.425 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.425 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.337317+0000", 2023-11-01T04:16:42.425 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0533000000000001e-05 2023-11-01T04:16:42.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.426 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.426 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.426 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.339886+0000", 2023-11-01T04:16:42.426 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002569034 2023-11-01T04:16:42.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.427 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.427 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.427 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.339985+0000", 2023-11-01T04:16:42.427 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.8431000000000007e-05 2023-11-01T04:16:42.427 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.427 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.428 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.428 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.428 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.9s4 58/31 ECSubWrite(tid=63, reqid=osd.7.0:63, at_version=58'57, trim_to=0'0, roll_forward_to=57'53))", 2023-11-01T04:16:42.428 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:38.349078+0000", 2023-11-01T04:16:42.428 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 3.8948077589999999, 2023-11-01T04:16:42.429 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20288057700000001, 2023-11-01T04:16:42.429 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.429 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.429 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.429 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.430 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.430 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.349078+0000", 2023-11-01T04:16:42.430 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.430 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.430 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.430 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.431 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.349078+0000", 2023-11-01T04:16:42.431 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.431 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.431 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.431 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.431 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.349080+0000", 2023-11-01T04:16:42.432 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.53e-06 2023-11-01T04:16:42.432 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.432 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.432 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.432 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.349136+0000", 2023-11-01T04:16:42.432 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5844999999999998e-05 2023-11-01T04:16:42.433 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.433 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.433 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.433 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.349166+0000", 2023-11-01T04:16:42.433 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0006999999999999e-05 2023-11-01T04:16:42.433 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.434 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.434 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.434 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.449262+0000", 2023-11-01T04:16:42.434 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10009615400000001 2023-11-01T04:16:42.434 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.434 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.435 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.435 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.549470+0000", 2023-11-01T04:16:42.435 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100208089 2023-11-01T04:16:42.435 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.435 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.435 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.436 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.549485+0000", 2023-11-01T04:16:42.436 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5150000000000001e-05 2023-11-01T04:16:42.436 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.436 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.436 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.436 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.551854+0000", 2023-11-01T04:16:42.437 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0023693910000000002 2023-11-01T04:16:42.437 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.437 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.437 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.437 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.551959+0000", 2023-11-01T04:16:42.437 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000104411 2023-11-01T04:16:42.438 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.438 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.438 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.438 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.438 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.439 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.0s2 59/31 ECSubRead(tid=26, to_read={3:0e42a959:::smithi19039307-46:head=0,192512,0}, subchunks={3:0e42a959:::smithi19039307-46:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:42.439 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:39.347572+0000", 2023-11-01T04:16:42.439 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 2.896314217, 2023-11-01T04:16:42.439 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10278812800000001, 2023-11-01T04:16:42.439 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.439 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.440 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.440 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.440 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.440 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:39.347572+0000", 2023-11-01T04:16:42.440 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.440 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.441 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.441 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.441 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:39.347572+0000", 2023-11-01T04:16:42.441 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.441 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.441 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.442 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.442 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:39.347573+0000", 2023-11-01T04:16:42.442 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3659999999999999e-06 2023-11-01T04:16:42.442 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.442 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.443 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.443 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:39.347590+0000", 2023-11-01T04:16:42.443 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6983e-05 2023-11-01T04:16:42.443 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.443 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.443 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.443 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:39.347648+0000", 2023-11-01T04:16:42.444 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7878999999999999e-05 2023-11-01T04:16:42.444 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.444 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.444 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.444 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:39.447781+0000", 2023-11-01T04:16:42.444 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10013272300000001 2023-11-01T04:16:42.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.445 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.445 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.445 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:39.450162+0000", 2023-11-01T04:16:42.445 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0023813609999999998 2023-11-01T04:16:42.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.446 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.446 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.446 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:39.450360+0000", 2023-11-01T04:16:42.446 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00019781599999999999 2023-11-01T04:16:42.446 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.446 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.447 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.447 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.447 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.447 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:42.448 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:42.449 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-01T04:16:42.449 INFO:tasks.rados.rados.0.smithi190.stdout:199: writing smithi19039307-16 from 3031040 to 3702784 tid 1 2023-11-01T04:16:42.449 INFO:tasks.rados.rados.0.smithi190.stdout:199: writing smithi19039307-16 from 3702784 to 4472832 tid 2 2023-11-01T04:16:42.449 INFO:tasks.rados.rados.0.smithi190.stdout:200: snap_create 2023-11-01T04:16:42.450 INFO:tasks.rados.rados.0.smithi190.stdout:194: finishing copy_from racing read to smithi19039307-17 2023-11-01T04:16:42.450 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-01T04:16:42.451 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:42.451 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-01T04:16:42.452 INFO:teuthology.orchestra.run.smithi153.stdout:osd.12: {} 2023-11-01T04:16:42.452 INFO:teuthology.orchestra.run.smithi153.stderr:osd.12: osd_enable_op_tracker = 'true' 2023-11-01T04:16:42.452 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:42.453 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-01T04:16:42.453 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:42.453 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-01T04:16:42.454 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:42.445+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:42.454 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:42.455 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-01T04:16:42.490 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 39 v 32 (ObjNum 66 snap 6 seq_num 66) dirty exists 2023-11-01T04:16:42.502 INFO:tasks.rados.rados.0.smithi190.stdout:194: finishing copy_from to smithi19039307-17 2023-11-01T04:16:42.502 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 17 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-01T04:16:42.549 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:42.546+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:42.614 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:42.615 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:16:42.615 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2023-11-01T04:16:42.615 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:42.629 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-01T04:16:42.668 INFO:tasks.rados.rados.0.smithi190.stdout:199: finishing write tid 1 to smithi19039307-16 2023-11-01T04:16:42.669 INFO:tasks.rados.rados.0.smithi190.stdout:199: finishing write tid 2 to smithi19039307-16 2023-11-01T04:16:42.669 INFO:tasks.rados.rados.0.smithi190.stdout:199: finishing write tid 3 to smithi19039307-16 2023-11-01T04:16:42.675 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:42.676+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:42.717 INFO:tasks.rados.rados.0.smithi190.stdout:196: finishing write tid 1 to smithi19039307-43 2023-11-01T04:16:42.750 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:42.750 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:16:42.751 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2023-11-01T04:16:42.751 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2023-11-01T04:16:42.751 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:42.751 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:42.751+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:42.765 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-01T04:16:42.851 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:42.852+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:42.915 INFO:tasks.rados.rados.0.smithi190.stdout:196: finishing write tid 2 to smithi19039307-43 2023-11-01T04:16:42.933 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:42.934 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2023-11-01T04:16:42.934 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2023-11-01T04:16:42.934 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:16:42.934 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.934 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=186, reqid=client.4559.0:156, at_version=36'13, trim_to=0'0, roll_forward_to=35'12))", 2023-11-01T04:16:42.935 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.399459+0000", 2023-11-01T04:16:42.935 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 27.532085016, 2023-11-01T04:16:42.935 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.102602861, 2023-11-01T04:16:42.935 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.935 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.936 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.936 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.936 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.936 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399459+0000", 2023-11-01T04:16:42.936 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.937 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.937 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.937 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399459+0000", 2023-11-01T04:16:42.937 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.938 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.938 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.938 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399460+0000", 2023-11-01T04:16:42.938 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3480000000000001e-06 2023-11-01T04:16:42.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.939 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.939 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.939 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399618+0000", 2023-11-01T04:16:42.939 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000158383 2023-11-01T04:16:42.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.940 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.940 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.940 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399635+0000", 2023-11-01T04:16:42.940 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6205000000000002e-05 2023-11-01T04:16:42.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.941 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.941 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.941 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399646+0000", 2023-11-01T04:16:42.941 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1274000000000001e-05 2023-11-01T04:16:42.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.942 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.942 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.942 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.499871+0000", 2023-11-01T04:16:42.942 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10022505 2023-11-01T04:16:42.942 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.943 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.943 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.943 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.499887+0000", 2023-11-01T04:16:42.943 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5743000000000002e-05 2023-11-01T04:16:42.943 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.944 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.944 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.944 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502013+0000", 2023-11-01T04:16:42.944 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0021257390000000002 2023-11-01T04:16:42.944 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.945 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.945 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.945 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502062+0000", 2023-11-01T04:16:42.945 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9119000000000002e-05 2023-11-01T04:16:42.945 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.946 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.946 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.946 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.946 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.946 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/31 ECSubWrite(tid=187, reqid=client.4559.0:157, at_version=36'14, trim_to=0'0, roll_forward_to=35'12))", 2023-11-01T04:16:42.946 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.399671+0000", 2023-11-01T04:16:42.947 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 27.53187294, 2023-11-01T04:16:42.947 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10242767899999999, 2023-11-01T04:16:42.947 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.947 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.947 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.948 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.948 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.948 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399671+0000", 2023-11-01T04:16:42.948 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.949 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.949 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.949 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.949 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399671+0000", 2023-11-01T04:16:42.949 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.949 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.950 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.950 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.950 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399671+0000", 2023-11-01T04:16:42.950 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2600000000000002e-07 2023-11-01T04:16:42.950 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.951 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.951 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.951 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399753+0000", 2023-11-01T04:16:42.951 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1821000000000001e-05 2023-11-01T04:16:42.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.952 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.952 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.952 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399761+0000", 2023-11-01T04:16:42.952 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.79e-06 2023-11-01T04:16:42.952 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.952 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.953 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.953 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399766+0000", 2023-11-01T04:16:42.953 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2639999999999999e-06 2023-11-01T04:16:42.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.953 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.954 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.954 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.500895+0000", 2023-11-01T04:16:42.954 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101128433 2023-11-01T04:16:42.954 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.954 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.955 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.955 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.500905+0000", 2023-11-01T04:16:42.955 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0156999999999999e-05 2023-11-01T04:16:42.955 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.955 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.955 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.956 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502071+0000", 2023-11-01T04:16:42.956 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001166032 2023-11-01T04:16:42.956 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.956 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.956 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.957 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.502099+0000", 2023-11-01T04:16:42.957 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7656000000000001e-05 2023-11-01T04:16:42.957 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.957 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.957 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.957 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.958 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.958 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.3s1 36/31 ECSubWrite(tid=189, reqid=client.4559.0:151, at_version=36'6, trim_to=0'0, roll_forward_to=35'4))", 2023-11-01T04:16:42.958 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.417087+0000", 2023-11-01T04:16:42.958 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 27.514456932000002, 2023-11-01T04:16:42.958 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103983992, 2023-11-01T04:16:42.959 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.959 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.959 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.959 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.959 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.960 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417087+0000", 2023-11-01T04:16:42.960 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.960 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.960 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.960 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.961 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417087+0000", 2023-11-01T04:16:42.961 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.961 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.961 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.962 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417087+0000", 2023-11-01T04:16:42.962 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2499999999999995e-07 2023-11-01T04:16:42.962 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.962 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.962 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.962 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417161+0000", 2023-11-01T04:16:42.963 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3508000000000002e-05 2023-11-01T04:16:42.963 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.963 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.963 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.963 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417171+0000", 2023-11-01T04:16:42.964 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7010000000000008e-06 2023-11-01T04:16:42.964 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.964 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.964 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.964 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417180+0000", 2023-11-01T04:16:42.964 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2140000000000002e-06 2023-11-01T04:16:42.965 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.965 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.965 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.965 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.517702+0000", 2023-11-01T04:16:42.965 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100522186 2023-11-01T04:16:42.966 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.966 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.966 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.966 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.517709+0000", 2023-11-01T04:16:42.966 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0299999999999996e-06 2023-11-01T04:16:42.967 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.967 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.967 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.967 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.521045+0000", 2023-11-01T04:16:42.967 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0033360640000000001 2023-11-01T04:16:42.967 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.968 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.968 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.968 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.521071+0000", 2023-11-01T04:16:42.968 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5763999999999999e-05 2023-11-01T04:16:42.968 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.968 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.969 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.969 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.969 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.969 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.3s1 36/31 ECSubWrite(tid=190, reqid=client.4559.0:152, at_version=36'7, trim_to=0'0, roll_forward_to=35'4))", 2023-11-01T04:16:42.969 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.417198+0000", 2023-11-01T04:16:42.970 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 27.514345552000002, 2023-11-01T04:16:42.970 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103890102, 2023-11-01T04:16:42.970 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.970 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.970 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.971 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.971 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.971 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417198+0000", 2023-11-01T04:16:42.971 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.971 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.971 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.972 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.972 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417198+0000", 2023-11-01T04:16:42.972 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.972 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.972 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.973 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.973 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417199+0000", 2023-11-01T04:16:42.973 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1300000000000001e-07 2023-11-01T04:16:42.973 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.973 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.973 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.974 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417214+0000", 2023-11-01T04:16:42.974 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5702999999999999e-05 2023-11-01T04:16:42.974 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.974 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.974 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.975 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417226+0000", 2023-11-01T04:16:42.975 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1602e-05 2023-11-01T04:16:42.975 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.975 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.975 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.976 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417238+0000", 2023-11-01T04:16:42.976 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2133000000000001e-05 2023-11-01T04:16:42.976 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.976 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.976 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.976 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.518027+0000", 2023-11-01T04:16:42.977 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100788603 2023-11-01T04:16:42.977 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.977 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.977 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.977 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.518038+0000", 2023-11-01T04:16:42.977 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0963e-05 2023-11-01T04:16:42.978 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.978 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.978 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.978 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.521074+0000", 2023-11-01T04:16:42.978 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0030366820000000002 2023-11-01T04:16:42.979 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.979 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.979 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.979 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.521088+0000", 2023-11-01T04:16:42.979 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4003e-05 2023-11-01T04:16:42.979 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.980 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.980 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.980 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.980 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.980 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.as6 36/31 ECSubWrite(tid=42, reqid=client.4559.0:167, at_version=0'0, trim_to=0'0, roll_forward_to=36'5))", 2023-11-01T04:16:42.981 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.729282+0000", 2023-11-01T04:16:42.981 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 26.202261460999999, 2023-11-01T04:16:42.981 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10185688399999999, 2023-11-01T04:16:42.981 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.981 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.981 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.982 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.982 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.982 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.729282+0000", 2023-11-01T04:16:42.982 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.982 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.983 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.983 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.983 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.729282+0000", 2023-11-01T04:16:42.983 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.983 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.983 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.984 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.984 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.729285+0000", 2023-11-01T04:16:42.984 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1900000000000002e-06 2023-11-01T04:16:42.984 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.984 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.985 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.985 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.729322+0000", 2023-11-01T04:16:42.985 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.7092000000000003e-05 2023-11-01T04:16:42.985 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.985 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.986 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.986 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.729368+0000", 2023-11-01T04:16:42.986 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6422999999999999e-05 2023-11-01T04:16:42.986 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.986 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.987 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.987 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.829469+0000", 2023-11-01T04:16:42.987 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10010103300000001 2023-11-01T04:16:42.987 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.987 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.987 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.988 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.829610+0000", 2023-11-01T04:16:42.988 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014110199999999999 2023-11-01T04:16:42.988 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.988 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.988 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:42.989 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.829633+0000", 2023-11-01T04:16:42.989 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3303e-05 2023-11-01T04:16:42.989 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.989 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.989 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:42.990 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.831049+0000", 2023-11-01T04:16:42.990 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001415794 2023-11-01T04:16:42.990 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.990 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.990 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:42.991 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.831139+0000", 2023-11-01T04:16:42.991 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9946999999999998e-05 2023-11-01T04:16:42.991 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.991 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:42.991 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:42.991 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.992 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.992 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.3s1 37/31 ECSubRead(tid=200, to_read={3:c78c701d:::smithi19039307-10:head=0,40960,0}, subchunks={3:c78c701d:::smithi19039307-10:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:42.992 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.967162+0000", 2023-11-01T04:16:42.992 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 25.964382035, 2023-11-01T04:16:42.992 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10201631, 2023-11-01T04:16:42.992 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:42.993 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:42.993 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:42.993 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.993 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:42.993 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967162+0000", 2023-11-01T04:16:42.994 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.994 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.994 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.994 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:42.994 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967162+0000", 2023-11-01T04:16:42.994 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:42.995 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.995 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.995 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:42.995 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967162+0000", 2023-11-01T04:16:42.995 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4099999999999999e-07 2023-11-01T04:16:42.996 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.996 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.996 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:42.996 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967172+0000", 2023-11-01T04:16:42.996 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4700000000000008e-06 2023-11-01T04:16:42.996 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.997 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.997 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:42.997 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967196+0000", 2023-11-01T04:16:42.997 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4190999999999999e-05 2023-11-01T04:16:42.997 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.997 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.998 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:42.998 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.967208+0000", 2023-11-01T04:16:42.998 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1973e-05 2023-11-01T04:16:42.998 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.998 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.999 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:42.999 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.068708+0000", 2023-11-01T04:16:42.999 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10149997299999999 2023-11-01T04:16:42.999 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:42.999 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:42.999 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.000 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.069178+0000", 2023-11-01T04:16:43.000 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00047026199999999999 2023-11-01T04:16:43.000 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.000 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.000 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.001 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.001 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.001 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 37/31 ECSubWrite(tid=44, reqid=client.4559.0:189, at_version=37'19, trim_to=0'0, roll_forward_to=36'17))", 2023-11-01T04:16:43.001 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:17.267116+0000", 2023-11-01T04:16:43.001 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 25.664428207, 2023-11-01T04:16:43.001 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103359019, 2023-11-01T04:16:43.002 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.002 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.002 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.002 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.002 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.002 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267116+0000", 2023-11-01T04:16:43.003 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.003 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.003 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.003 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.003 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267116+0000", 2023-11-01T04:16:43.004 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.004 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.004 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.004 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.004 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267118+0000", 2023-11-01T04:16:43.004 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0729999999999999e-06 2023-11-01T04:16:43.005 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.005 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.005 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.005 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267268+0000", 2023-11-01T04:16:43.005 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015054599999999999 2023-11-01T04:16:43.005 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.006 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.006 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.006 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267309+0000", 2023-11-01T04:16:43.006 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1187999999999999e-05 2023-11-01T04:16:43.006 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.007 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.007 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.007 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.267336+0000", 2023-11-01T04:16:43.007 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7093e-05 2023-11-01T04:16:43.007 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.007 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.008 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.008 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.367659+0000", 2023-11-01T04:16:43.008 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10032273799999999 2023-11-01T04:16:43.008 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.008 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.009 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.009 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.367685+0000", 2023-11-01T04:16:43.009 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5454999999999999e-05 2023-11-01T04:16:43.009 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.009 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.009 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.010 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.370407+0000", 2023-11-01T04:16:43.010 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0027224979999999998 2023-11-01T04:16:43.010 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.010 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.010 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.011 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.370475+0000", 2023-11-01T04:16:43.011 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7428000000000006e-05 2023-11-01T04:16:43.011 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.011 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.011 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.011 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.012 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.012 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s4 37/31 ECSubWrite(tid=207, reqid=client.4559.0:207, at_version=37'21, trim_to=0'0, roll_forward_to=36'16))", 2023-11-01T04:16:43.012 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:17.382276+0000", 2023-11-01T04:16:43.012 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 25.549267429, 2023-11-01T04:16:43.012 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103388864, 2023-11-01T04:16:43.012 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.013 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.013 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.013 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.013 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.013 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.382276+0000", 2023-11-01T04:16:43.014 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.014 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.014 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.014 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.014 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.382276+0000", 2023-11-01T04:16:43.015 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.015 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.015 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.015 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.015 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.382278+0000", 2023-11-01T04:16:43.015 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7630000000000001e-06 2023-11-01T04:16:43.016 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.016 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.016 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.016 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.382534+0000", 2023-11-01T04:16:43.016 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00025576899999999998 2023-11-01T04:16:43.016 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.017 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.017 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.017 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.382567+0000", 2023-11-01T04:16:43.017 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2719000000000003e-05 2023-11-01T04:16:43.017 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.018 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.018 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.018 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.482664+0000", 2023-11-01T04:16:43.018 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100097424 2023-11-01T04:16:43.018 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.018 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.019 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.019 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.482806+0000", 2023-11-01T04:16:43.019 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014185599999999999 2023-11-01T04:16:43.019 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.019 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.019 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.020 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.482822+0000", 2023-11-01T04:16:43.020 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5743000000000002e-05 2023-11-01T04:16:43.020 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.020 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.020 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.021 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.485590+0000", 2023-11-01T04:16:43.021 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0027687990000000002 2023-11-01T04:16:43.021 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.021 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.021 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.021 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.485665+0000", 2023-11-01T04:16:43.022 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4790999999999996e-05 2023-11-01T04:16:43.022 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.022 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.022 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.022 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.022 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.023 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 41/31 ECSubWrite(tid=55, reqid=client.4559.0:266, at_version=41'21, trim_to=0'0, roll_forward_to=37'19))", 2023-11-01T04:16:43.023 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.119591+0000", 2023-11-01T04:16:43.023 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 21.811952444999999, 2023-11-01T04:16:43.023 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.106535935, 2023-11-01T04:16:43.023 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.023 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.024 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.024 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.024 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.024 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.119591+0000", 2023-11-01T04:16:43.024 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.024 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.025 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.025 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.025 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.119591+0000", 2023-11-01T04:16:43.025 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.025 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.025 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.026 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.026 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.119593+0000", 2023-11-01T04:16:43.026 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9230000000000001e-06 2023-11-01T04:16:43.026 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.026 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.027 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.027 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.119864+0000", 2023-11-01T04:16:43.027 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000270623 2023-11-01T04:16:43.027 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.027 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.027 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.028 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.119910+0000", 2023-11-01T04:16:43.028 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6144999999999999e-05 2023-11-01T04:16:43.028 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.028 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.028 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.028 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.119934+0000", 2023-11-01T04:16:43.029 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3643e-05 2023-11-01T04:16:43.029 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.029 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.029 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.029 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.120148+0000", 2023-11-01T04:16:43.029 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000214224 2023-11-01T04:16:43.030 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.030 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.030 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.030 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.120173+0000", 2023-11-01T04:16:43.030 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4790000000000002e-05 2023-11-01T04:16:43.030 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.031 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.031 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.031 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.226085+0000", 2023-11-01T04:16:43.031 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10591191699999999 2023-11-01T04:16:43.031 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.031 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.032 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.032 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.226127+0000", 2023-11-01T04:16:43.032 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2670000000000003e-05 2023-11-01T04:16:43.032 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.032 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.032 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.032 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.033 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.033 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 41/31 ECSubWrite(tid=56, reqid=client.4559.0:267, at_version=41'22, trim_to=0'0, roll_forward_to=37'19))", 2023-11-01T04:16:43.033 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.121320+0000", 2023-11-01T04:16:43.033 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 21.810223820000001, 2023-11-01T04:16:43.033 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104845891, 2023-11-01T04:16:43.033 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.034 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.034 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.034 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.034 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.034 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.121320+0000", 2023-11-01T04:16:43.035 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.035 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.035 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.035 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.035 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.121320+0000", 2023-11-01T04:16:43.035 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.036 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.036 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.036 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.036 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.121322+0000", 2023-11-01T04:16:43.036 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0360000000000001e-06 2023-11-01T04:16:43.036 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.037 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.037 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.037 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.121511+0000", 2023-11-01T04:16:43.037 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000188612 2023-11-01T04:16:43.037 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.037 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.038 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.038 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.121545+0000", 2023-11-01T04:16:43.038 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4269e-05 2023-11-01T04:16:43.038 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.038 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.038 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.039 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.121571+0000", 2023-11-01T04:16:43.039 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6043000000000001e-05 2023-11-01T04:16:43.039 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.039 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.039 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.039 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122494+0000", 2023-11-01T04:16:43.040 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00092303199999999995 2023-11-01T04:16:43.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.040 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.040 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.040 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122518+0000", 2023-11-01T04:16:43.040 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4036000000000002e-05 2023-11-01T04:16:43.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.041 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.041 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.041 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.226144+0000", 2023-11-01T04:16:43.041 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103625515 2023-11-01T04:16:43.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.042 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.042 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.042 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.226166+0000", 2023-11-01T04:16:43.042 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2348e-05 2023-11-01T04:16:43.042 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.042 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.043 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.043 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.043 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.043 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 41/31 ECSubWrite(tid=57, reqid=client.4559.0:268, at_version=41'23, trim_to=0'0, roll_forward_to=37'19))", 2023-11-01T04:16:43.043 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.122597+0000", 2023-11-01T04:16:43.043 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 21.808946867, 2023-11-01T04:16:43.044 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.106843242, 2023-11-01T04:16:43.044 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.044 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.044 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.044 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.044 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.045 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122597+0000", 2023-11-01T04:16:43.045 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.045 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.045 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.045 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.045 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122597+0000", 2023-11-01T04:16:43.046 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.046 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.046 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.046 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.046 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122599+0000", 2023-11-01T04:16:43.047 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6729999999999999e-06 2023-11-01T04:16:43.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.047 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.047 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.047 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122640+0000", 2023-11-01T04:16:43.047 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1768999999999999e-05 2023-11-01T04:16:43.048 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.048 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.048 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.048 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122673+0000", 2023-11-01T04:16:43.048 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2202000000000001e-05 2023-11-01T04:16:43.048 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.049 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.049 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.049 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122695+0000", 2023-11-01T04:16:43.049 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2657e-05 2023-11-01T04:16:43.049 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.049 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.050 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.050 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.224915+0000", 2023-11-01T04:16:43.050 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10221991599999999 2023-11-01T04:16:43.050 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.050 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.050 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.051 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.224950+0000", 2023-11-01T04:16:43.051 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4789000000000002e-05 2023-11-01T04:16:43.051 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.051 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.051 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.051 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.229406+0000", 2023-11-01T04:16:43.052 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0044560440000000002 2023-11-01T04:16:43.052 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.052 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.052 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.052 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.229440+0000", 2023-11-01T04:16:43.052 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4192000000000001e-05 2023-11-01T04:16:43.053 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.053 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.053 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.053 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.053 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.053 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 41/31 ECSubWrite(tid=58, reqid=osd.12.0:58, at_version=41'25, trim_to=0'0, roll_forward_to=37'19))", 2023-11-01T04:16:43.054 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.123645+0000", 2023-11-01T04:16:43.054 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 21.807898494, 2023-11-01T04:16:43.054 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.105819151, 2023-11-01T04:16:43.054 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.054 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.054 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.055 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.055 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.055 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123645+0000", 2023-11-01T04:16:43.055 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.055 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.055 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.056 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.056 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123645+0000", 2023-11-01T04:16:43.056 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.056 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.056 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.056 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.057 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123647+0000", 2023-11-01T04:16:43.057 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.716e-06 2023-11-01T04:16:43.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.057 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.057 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.057 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123698+0000", 2023-11-01T04:16:43.058 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1474999999999998e-05 2023-11-01T04:16:43.058 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.058 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.058 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.058 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123730+0000", 2023-11-01T04:16:43.058 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1600000000000002e-05 2023-11-01T04:16:43.059 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.059 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.059 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.059 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.223820+0000", 2023-11-01T04:16:43.059 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100089995 2023-11-01T04:16:43.060 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.060 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.060 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.060 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.225389+0000", 2023-11-01T04:16:43.060 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0015688150000000001 2023-11-01T04:16:43.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.061 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.061 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.061 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.225407+0000", 2023-11-01T04:16:43.061 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7980000000000001e-05 2023-11-01T04:16:43.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.062 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.062 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.062 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.229444+0000", 2023-11-01T04:16:43.062 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0040372419999999999 2023-11-01T04:16:43.062 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.062 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.063 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.063 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.229464+0000", 2023-11-01T04:16:43.063 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0327999999999999e-05 2023-11-01T04:16:43.063 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.063 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.063 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.064 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.064 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.064 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s4 46/31 ECSubWrite(tid=236, reqid=client.4559.0:330, at_version=46'36, trim_to=0'0, roll_forward_to=45'33))", 2023-11-01T04:16:43.064 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.282877+0000", 2023-11-01T04:16:43.064 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 16.648666374000001, 2023-11-01T04:16:43.064 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20107656700000001, 2023-11-01T04:16:43.065 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.065 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.065 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.065 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.065 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.065 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282877+0000", 2023-11-01T04:16:43.066 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.066 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.066 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.066 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.066 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282877+0000", 2023-11-01T04:16:43.066 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.067 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.067 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.067 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.067 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282880+0000", 2023-11-01T04:16:43.067 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.357e-06 2023-11-01T04:16:43.067 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.068 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.068 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.068 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.283033+0000", 2023-11-01T04:16:43.068 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015317900000000001 2023-11-01T04:16:43.068 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.068 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.069 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.069 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.283074+0000", 2023-11-01T04:16:43.069 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0942999999999997e-05 2023-11-01T04:16:43.069 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.069 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.069 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.070 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383201+0000", 2023-11-01T04:16:43.070 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100126696 2023-11-01T04:16:43.070 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.070 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.070 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.071 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383268+0000", 2023-11-01T04:16:43.071 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7747e-05 2023-11-01T04:16:43.071 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.071 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.071 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.071 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383280+0000", 2023-11-01T04:16:43.071 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1242e-05 2023-11-01T04:16:43.072 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.072 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.072 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.072 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.483899+0000", 2023-11-01T04:16:43.072 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10061980399999999 2023-11-01T04:16:43.072 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.073 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.073 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.073 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.483954+0000", 2023-11-01T04:16:43.073 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4598999999999997e-05 2023-11-01T04:16:43.073 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.073 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.074 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.074 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.074 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.074 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s4 48/31 ECSubWrite(tid=250, reqid=client.4559.0:378, at_version=0'0, trim_to=0'0, roll_forward_to=48'39))", 2023-11-01T04:16:43.074 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:28.574447+0000", 2023-11-01T04:16:43.075 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 14.357096417999999, 2023-11-01T04:16:43.075 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101895423, 2023-11-01T04:16:43.075 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.075 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.075 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.075 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.075 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.076 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.574447+0000", 2023-11-01T04:16:43.076 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.076 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.076 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.076 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.077 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.574447+0000", 2023-11-01T04:16:43.077 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.077 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.077 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.077 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.077 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.574450+0000", 2023-11-01T04:16:43.077 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2060000000000001e-06 2023-11-01T04:16:43.078 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.078 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.078 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.078 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.574473+0000", 2023-11-01T04:16:43.078 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3794e-05 2023-11-01T04:16:43.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.079 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.079 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.079 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.574513+0000", 2023-11-01T04:16:43.079 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9745e-05 2023-11-01T04:16:43.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.080 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.080 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.080 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.674654+0000", 2023-11-01T04:16:43.080 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100140721 2023-11-01T04:16:43.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.080 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.081 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.081 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.674877+0000", 2023-11-01T04:16:43.081 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00022327100000000001 2023-11-01T04:16:43.081 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.081 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.081 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.082 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.674911+0000", 2023-11-01T04:16:43.082 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.341e-05 2023-11-01T04:16:43.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.082 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.082 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.082 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.676258+0000", 2023-11-01T04:16:43.083 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001347041 2023-11-01T04:16:43.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.083 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.083 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.083 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.676343+0000", 2023-11-01T04:16:43.083 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5235000000000006e-05 2023-11-01T04:16:43.084 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.084 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.084 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.084 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.084 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 57/31 ECSubWrite(tid=103, reqid=osd.12.0:103, at_version=57'54, trim_to=0'0, roll_forward_to=55'52))", 2023-11-01T04:16:43.085 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:37.236006+0000", 2023-11-01T04:16:43.085 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.6955378469999998, 2023-11-01T04:16:43.085 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10216819100000001, 2023-11-01T04:16:43.085 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.086 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.086 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.086 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.086 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.086 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.236006+0000", 2023-11-01T04:16:43.087 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.087 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.087 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.087 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.087 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.236006+0000", 2023-11-01T04:16:43.088 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.088 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.088 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.088 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.088 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.236006+0000", 2023-11-01T04:16:43.089 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.32e-07 2023-11-01T04:16:43.089 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.089 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.089 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.090 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.236050+0000", 2023-11-01T04:16:43.090 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3365000000000002e-05 2023-11-01T04:16:43.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.090 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.090 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.091 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.236069+0000", 2023-11-01T04:16:43.091 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9349999999999999e-05 2023-11-01T04:16:43.091 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.091 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.091 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.091 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.236081+0000", 2023-11-01T04:16:43.092 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2364999999999999e-05 2023-11-01T04:16:43.092 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.092 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.092 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.092 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.336238+0000", 2023-11-01T04:16:43.092 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10015671399999999 2023-11-01T04:16:43.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.093 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.093 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.093 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.336250+0000", 2023-11-01T04:16:43.093 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2130000000000001e-05 2023-11-01T04:16:43.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.094 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.094 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.094 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.338091+0000", 2023-11-01T04:16:43.094 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0018402340000000001 2023-11-01T04:16:43.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.095 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.095 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.095 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.338174+0000", 2023-11-01T04:16:43.095 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3601000000000001e-05 2023-11-01T04:16:43.095 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.095 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.096 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.096 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.096 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 57/31 ECSubWrite(tid=104, reqid=osd.12.0:104, at_version=57'56, trim_to=0'0, roll_forward_to=55'52))", 2023-11-01T04:16:43.096 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:37.237023+0000", 2023-11-01T04:16:43.096 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.6945208599999999, 2023-11-01T04:16:43.097 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10307158, 2023-11-01T04:16:43.097 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.097 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.097 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.097 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.097 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.098 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.237023+0000", 2023-11-01T04:16:43.098 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.098 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.098 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.099 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.237023+0000", 2023-11-01T04:16:43.099 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.099 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.099 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.099 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.099 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.237024+0000", 2023-11-01T04:16:43.100 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5300000000000002e-07 2023-11-01T04:16:43.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.100 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.100 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.100 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.237053+0000", 2023-11-01T04:16:43.100 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9354000000000001e-05 2023-11-01T04:16:43.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.101 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.101 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.101 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.237074+0000", 2023-11-01T04:16:43.101 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0468000000000001e-05 2023-11-01T04:16:43.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.101 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.102 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.102 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.237087+0000", 2023-11-01T04:16:43.102 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2981e-05 2023-11-01T04:16:43.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.102 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.103 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.103 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.336765+0000", 2023-11-01T04:16:43.103 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.099678644999999996 2023-11-01T04:16:43.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.103 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.103 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.104 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.336792+0000", 2023-11-01T04:16:43.104 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6687e-05 2023-11-01T04:16:43.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.104 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.104 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.104 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.340029+0000", 2023-11-01T04:16:43.105 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0032373409999999999 2023-11-01T04:16:43.105 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.105 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.105 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.105 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.340095+0000", 2023-11-01T04:16:43.105 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5151e-05 2023-11-01T04:16:43.106 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.106 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.106 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.106 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.106 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 57/31 ECSubWrite(tid=107, reqid=osd.12.0:106, at_version=0'0, trim_to=0'0, roll_forward_to=57'58))", 2023-11-01T04:16:43.107 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:37.836388+0000", 2023-11-01T04:16:43.107 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.0951562309999998, 2023-11-01T04:16:43.107 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.102039191, 2023-11-01T04:16:43.107 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.107 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.107 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.108 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.108 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.108 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.836388+0000", 2023-11-01T04:16:43.108 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.108 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.109 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.109 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.836388+0000", 2023-11-01T04:16:43.109 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.109 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.110 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.110 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.836389+0000", 2023-11-01T04:16:43.110 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6959999999999999e-06 2023-11-01T04:16:43.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.110 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.111 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.111 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.836416+0000", 2023-11-01T04:16:43.111 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6713e-05 2023-11-01T04:16:43.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.111 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.111 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.112 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.836447+0000", 2023-11-01T04:16:43.112 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1412999999999999e-05 2023-11-01T04:16:43.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.112 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.112 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.112 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.836471+0000", 2023-11-01T04:16:43.113 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3359000000000001e-05 2023-11-01T04:16:43.113 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.113 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.113 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.113 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.836659+0000", 2023-11-01T04:16:43.113 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000188557 2023-11-01T04:16:43.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.114 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.114 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.114 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.836691+0000", 2023-11-01T04:16:43.114 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1245999999999999e-05 2023-11-01T04:16:43.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.115 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.115 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.115 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.938280+0000", 2023-11-01T04:16:43.115 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10158937699999999 2023-11-01T04:16:43.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.116 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.116 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.116 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:37.938427+0000", 2023-11-01T04:16:43.116 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014683 2023-11-01T04:16:43.116 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.116 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.117 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.117 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.117 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.3s1 58/31 ECSubWrite(tid=279, reqid=osd.13.0:279, at_version=58'37, trim_to=0'0, roll_forward_to=57'33))", 2023-11-01T04:16:43.117 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:38.143156+0000", 2023-11-01T04:16:43.117 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 4.7883880379999999, 2023-11-01T04:16:43.118 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104186716, 2023-11-01T04:16:43.118 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.118 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.118 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.118 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.118 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.119 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.143156+0000", 2023-11-01T04:16:43.119 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.119 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.120 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.143156+0000", 2023-11-01T04:16:43.120 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.120 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.120 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.120 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.143157+0000", 2023-11-01T04:16:43.121 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.573e-06 2023-11-01T04:16:43.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.121 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.121 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.121 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.143208+0000", 2023-11-01T04:16:43.121 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0283e-05 2023-11-01T04:16:43.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.122 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.122 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.143237+0000", 2023-11-01T04:16:43.122 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9632999999999999e-05 2023-11-01T04:16:43.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.123 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.123 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.123 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243350+0000", 2023-11-01T04:16:43.123 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10011239399999999 2023-11-01T04:16:43.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.123 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.124 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.124 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243546+0000", 2023-11-01T04:16:43.124 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000196546 2023-11-01T04:16:43.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.124 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.124 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.124 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243574+0000", 2023-11-01T04:16:43.125 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7372000000000002e-05 2023-11-01T04:16:43.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.125 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.125 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.125 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.247245+0000", 2023-11-01T04:16:43.125 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0036718900000000001 2023-11-01T04:16:43.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.126 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.126 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.126 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.247342+0000", 2023-11-01T04:16:43.126 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7025e-05 2023-11-01T04:16:43.127 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.127 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.127 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.127 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.127 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 58/31 ECSubWrite(tid=109, reqid=osd.12.0:109, at_version=58'62, trim_to=0'0, roll_forward_to=57'58))", 2023-11-01T04:16:43.128 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:38.243889+0000", 2023-11-01T04:16:43.128 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 4.6876544429999996, 2023-11-01T04:16:43.128 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.105244037, 2023-11-01T04:16:43.128 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.128 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.128 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.129 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.129 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243889+0000", 2023-11-01T04:16:43.129 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.129 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.130 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.130 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243889+0000", 2023-11-01T04:16:43.130 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.130 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.130 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.131 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243890+0000", 2023-11-01T04:16:43.131 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0100000000000001e-06 2023-11-01T04:16:43.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.131 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.131 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.131 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243930+0000", 2023-11-01T04:16:43.132 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0000000000000003e-05 2023-11-01T04:16:43.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.132 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.132 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.132 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243945+0000", 2023-11-01T04:16:43.133 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4473e-05 2023-11-01T04:16:43.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.133 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.133 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.133 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243959+0000", 2023-11-01T04:16:43.133 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3763e-05 2023-11-01T04:16:43.134 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.134 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.134 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.134 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.245005+0000", 2023-11-01T04:16:43.134 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00104667 2023-11-01T04:16:43.134 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.135 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.135 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.135 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.245035+0000", 2023-11-01T04:16:43.135 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9411999999999999e-05 2023-11-01T04:16:43.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.135 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.136 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.136 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.349054+0000", 2023-11-01T04:16:43.136 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10401964499999999 2023-11-01T04:16:43.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.136 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.136 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.137 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.349133+0000", 2023-11-01T04:16:43.137 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9064000000000001e-05 2023-11-01T04:16:43.137 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.137 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.137 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.138 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.138 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 58/31 ECSubWrite(tid=111, reqid=osd.12.0:111, at_version=58'64, trim_to=0'0, roll_forward_to=58'62))", 2023-11-01T04:16:43.138 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:38.360098+0000", 2023-11-01T04:16:43.138 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 4.5714457790000003, 2023-11-01T04:16:43.138 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20292976700000001, 2023-11-01T04:16:43.139 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.139 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.139 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.139 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.139 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.360098+0000", 2023-11-01T04:16:43.140 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.140 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.140 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.140 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.360098+0000", 2023-11-01T04:16:43.140 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.141 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.141 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.141 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.360100+0000", 2023-11-01T04:16:43.141 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7799999999999999e-06 2023-11-01T04:16:43.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.142 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.142 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.360153+0000", 2023-11-01T04:16:43.142 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2722000000000001e-05 2023-11-01T04:16:43.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.143 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.143 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.143 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.360195+0000", 2023-11-01T04:16:43.143 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2905999999999999e-05 2023-11-01T04:16:43.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.143 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.144 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.144 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.460529+0000", 2023-11-01T04:16:43.144 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10033367 2023-11-01T04:16:43.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.144 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.144 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.144 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.560802+0000", 2023-11-01T04:16:43.145 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100272668 2023-11-01T04:16:43.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.145 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.145 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.145 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.560837+0000", 2023-11-01T04:16:43.145 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5589000000000001e-05 2023-11-01T04:16:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.562956+0000", 2023-11-01T04:16:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0021184530000000002 2023-11-01T04:16:43.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.563028+0000", 2023-11-01T04:16:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1978999999999996e-05 2023-11-01T04:16:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.147 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.148 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.148 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:43.150 INFO:tasks.rados.rados.0.smithi190.stdout:199: finishing write tid 4 to smithi19039307-16 2023-11-01T04:16:43.150 INFO:tasks.rados.rados.0.smithi190.stdout:199: oid 16 updating version 0 to 63 2023-11-01T04:16:43.150 INFO:tasks.rados.rados.0.smithi190.stdout:199: oid 16 updating version 63 to 64 2023-11-01T04:16:43.150 INFO:tasks.rados.rados.0.smithi190.stdout:199: oid 16 version 64 is already newer than 62 2023-11-01T04:16:43.150 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 16 v 64 (ObjNum 85 snap 17 seq_num 85) dirty exists 2023-11-01T04:16:43.151 INFO:tasks.rados.rados.0.smithi190.stdout:199: left oid 16 (ObjNum 85 snap 17 seq_num 85) 2023-11-01T04:16:43.151 INFO:tasks.rados.rados.0.smithi190.stdout:196: finishing write tid 3 to smithi19039307-43 2023-11-01T04:16:43.151 INFO:tasks.rados.rados.0.smithi190.stdout:196: finishing write tid 4 to smithi19039307-43 2023-11-01T04:16:43.151 INFO:tasks.rados.rados.0.smithi190.stdout:195: expect (ObjNum 53 snap 1 seq_num 53) 2023-11-01T04:16:43.151 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-01T04:16:43.152 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:43.148+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:43.187 INFO:teuthology.orchestra.run.smithi153.stderr:noscrub is set 2023-11-01T04:16:43.222 INFO:tasks.rados.rados.0.smithi190.stdout:194: done (6 left) 2023-11-01T04:16:43.222 INFO:tasks.rados.rados.0.smithi190.stdout:195: done (5 left) 2023-11-01T04:16:43.222 INFO:tasks.rados.rados.0.smithi190.stdout:197: done (4 left) 2023-11-01T04:16:43.222 INFO:tasks.rados.rados.0.smithi190.stdout:198: done (3 left) 2023-11-01T04:16:43.222 INFO:tasks.rados.rados.0.smithi190.stdout:199: done (2 left) 2023-11-01T04:16:43.223 INFO:tasks.rados.rados.0.smithi190.stdout:200: done (1 left) 2023-11-01T04:16:43.223 INFO:tasks.rados.rados.0.smithi190.stdout:201: snap_create 2023-11-01T04:16:43.250 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:43.249+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:43.309 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:43.309 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:16:43.309 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2023-11-01T04:16:43.310 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:43.323 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-01T04:16:43.323 INFO:tasks.rados.rados.0.smithi190.stdout:196: finishing write tid 5 to smithi19039307-43 2023-11-01T04:16:43.324 INFO:tasks.rados.rados.0.smithi190.stdout:196: oid 43 updating version 0 to 65 2023-11-01T04:16:43.324 INFO:tasks.rados.rados.0.smithi190.stdout:196: oid 43 updating version 65 to 66 2023-11-01T04:16:43.324 INFO:tasks.rados.rados.0.smithi190.stdout:196: oid 43 version 66 is already newer than 64 2023-11-01T04:16:43.324 INFO:tasks.rados.rados.0.smithi190.stdout:196: oid 43 version 66 is already newer than 63 2023-11-01T04:16:43.324 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 43 v 66 (ObjNum 84 snap 17 seq_num 84) dirty exists 2023-11-01T04:16:43.325 INFO:tasks.rados.rados.0.smithi190.stdout:196: left oid 43 (ObjNum 84 snap 17 seq_num 84) 2023-11-01T04:16:43.352 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:43.351+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:43.446 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:43.446 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:16:43.447 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2023-11-01T04:16:43.447 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2023-11-01T04:16:43.447 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:43.453 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:43.449+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:43.461 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-01T04:16:43.553 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:43.549+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:43.586 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:43.586 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2023-11-01T04:16:43.586 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2023-11-01T04:16:43.586 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:16:43.586 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.587 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:167 3.as0 3:5f19faeb:::smithi19039307-47:head [copy-from ver 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:43.587 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.317332+0000", 2023-11-01T04:16:43.587 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 27.266386692000001, 2023-11-01T04:16:43.587 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.411656464, 2023-11-01T04:16:43.587 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.588 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:43.588 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:43.588 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:43.588 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:43.588 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167 2023-11-01T04:16:43.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.589 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.589 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.589 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.589 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317332+0000", 2023-11-01T04:16:43.589 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.590 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.590 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.590 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317332+0000", 2023-11-01T04:16:43.590 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.591 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.591 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.591 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317334+0000", 2023-11-01T04:16:43.591 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.216e-06 2023-11-01T04:16:43.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.592 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.592 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.592 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317345+0000", 2023-11-01T04:16:43.592 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1381e-05 2023-11-01T04:16:43.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.593 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.593 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.593 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317372+0000", 2023-11-01T04:16:43.593 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6548000000000001e-05 2023-11-01T04:16:43.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.594 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.594 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.594 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317385+0000", 2023-11-01T04:16:43.594 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2514000000000001e-05 2023-11-01T04:16:43.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.594 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.595 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.595 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317457+0000", 2023-11-01T04:16:43.595 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2197e-05 2023-11-01T04:16:43.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.595 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.595 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:43.596 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317583+0000", 2023-11-01T04:16:43.596 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000126372 2023-11-01T04:16:43.596 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.596 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.596 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.597 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.524296+0000", 2023-11-01T04:16:43.597 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20671239599999999 2023-11-01T04:16:43.597 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.597 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.597 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.597 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.526421+0000", 2023-11-01T04:16:43.598 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0021255660000000002 2023-11-01T04:16:43.598 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.598 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.598 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:16:43.598 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.728850+0000", 2023-11-01T04:16:43.598 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.202429255 2023-11-01T04:16:43.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.599 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.599 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.599 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.728988+0000", 2023-11-01T04:16:43.599 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013801899999999999 2023-11-01T04:16:43.599 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.600 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.600 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.600 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.600 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:169 3.ds0 3:bc424c1c:::smithi19039307-35:head [copy-from ver 15 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:43.600 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.317475+0000", 2023-11-01T04:16:43.601 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 27.266243381999999, 2023-11-01T04:16:43.601 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.70963304599999999, 2023-11-01T04:16:43.601 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.601 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:43.601 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:43.602 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:43.602 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:43.602 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 169 2023-11-01T04:16:43.602 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.602 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.602 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.603 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.603 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317475+0000", 2023-11-01T04:16:43.603 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.603 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.603 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.603 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.604 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317475+0000", 2023-11-01T04:16:43.604 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.604 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.604 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.604 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317476+0000", 2023-11-01T04:16:43.605 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.026e-06 2023-11-01T04:16:43.605 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.605 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.605 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.605 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317482+0000", 2023-11-01T04:16:43.605 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6869999999999999e-06 2023-11-01T04:16:43.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.606 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.606 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.606 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.317494+0000", 2023-11-01T04:16:43.606 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2529e-05 2023-11-01T04:16:43.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.607 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.607 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.607 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417610+0000", 2023-11-01T04:16:43.607 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100115419 2023-11-01T04:16:43.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.608 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.608 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.608 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417723+0000", 2023-11-01T04:16:43.608 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011284499999999999 2023-11-01T04:16:43.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.608 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.609 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:43.609 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417815+0000", 2023-11-01T04:16:43.609 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2718000000000002e-05 2023-11-01T04:16:43.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.609 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.609 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.610 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.630806+0000", 2023-11-01T04:16:43.610 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.21299037200000001 2023-11-01T04:16:43.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.610 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.610 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.610 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.725358+0000", 2023-11-01T04:16:43.611 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.094552420999999998 2023-11-01T04:16:43.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.611 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.611 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:16:43.611 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.027005+0000", 2023-11-01T04:16:43.612 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30164710099999997 2023-11-01T04:16:43.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.612 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.612 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.612 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.027108+0000", 2023-11-01T04:16:43.612 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000102928 2023-11-01T04:16:43.613 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.613 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.613 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.613 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.613 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:170 3.ds0 3:bc424c1c:::smithi19039307-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:43.614 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.417683+0000", 2023-11-01T04:16:43.614 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 27.166035424, 2023-11-01T04:16:43.614 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30571221100000001, 2023-11-01T04:16:43.614 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.614 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:43.614 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:43.615 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:43.615 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:43.615 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 170 2023-11-01T04:16:43.615 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.615 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.616 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.616 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.616 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417683+0000", 2023-11-01T04:16:43.616 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.616 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.617 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.617 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417683+0000", 2023-11-01T04:16:43.617 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.617 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.617 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.618 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417685+0000", 2023-11-01T04:16:43.618 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.578e-06 2023-11-01T04:16:43.618 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.618 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.618 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.618 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417693+0000", 2023-11-01T04:16:43.619 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4810000000000006e-06 2023-11-01T04:16:43.619 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.619 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.619 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.619 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417706+0000", 2023-11-01T04:16:43.619 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2703e-05 2023-11-01T04:16:43.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.620 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.620 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.620 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.417714+0000", 2023-11-01T04:16:43.620 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6890000000000004e-06 2023-11-01T04:16:43.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.621 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.621 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.621 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.418000+0000", 2023-11-01T04:16:43.621 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00028649300000000002 2023-11-01T04:16:43.621 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.621 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.622 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for blocked object", 2023-11-01T04:16:43.622 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.418160+0000", 2023-11-01T04:16:43.622 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015998600000000001 2023-11-01T04:16:43.622 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.622 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.622 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.623 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.722848+0000", 2023-11-01T04:16:43.623 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30468759400000001 2023-11-01T04:16:43.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.623 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.623 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:43.624 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.723116+0000", 2023-11-01T04:16:43.624 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00026845200000000002 2023-11-01T04:16:43.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.624 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.624 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.624 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.723395+0000", 2023-11-01T04:16:43.624 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00027923500000000002 2023-11-01T04:16:43.625 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.625 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.625 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.625 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.625 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:173 3.ds0 3:b53d1bb4:::smithi19039307-22:head [read 0~360448,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:43.626 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.622283+0000", 2023-11-01T04:16:43.626 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 26.961435353999999, 2023-11-01T04:16:43.626 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.405767341, 2023-11-01T04:16:43.626 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.626 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:43.627 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:43.627 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:43.627 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:43.627 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 173 2023-11-01T04:16:43.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.627 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.628 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.628 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.628 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622283+0000", 2023-11-01T04:16:43.628 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.628 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.628 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.629 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622283+0000", 2023-11-01T04:16:43.629 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.629 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.629 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.629 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622284+0000", 2023-11-01T04:16:43.630 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7199999999999999e-07 2023-11-01T04:16:43.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.630 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.630 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.630 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622287+0000", 2023-11-01T04:16:43.630 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1879999999999998e-06 2023-11-01T04:16:43.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.631 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.631 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.631 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622295+0000", 2023-11-01T04:16:43.631 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.8820000000000001e-06 2023-11-01T04:16:43.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.632 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.632 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.632 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622300+0000", 2023-11-01T04:16:43.632 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9049999999999996e-06 2023-11-01T04:16:43.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.632 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.633 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.633 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622734+0000", 2023-11-01T04:16:43.633 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0004347 2023-11-01T04:16:43.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.633 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.633 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:43.634 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622866+0000", 2023-11-01T04:16:43.634 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013168300000000001 2023-11-01T04:16:43.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.634 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.634 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.634 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.028051+0000", 2023-11-01T04:16:43.635 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.40518451100000002 2023-11-01T04:16:43.635 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.635 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.635 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.636 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:174 3.ds0 3:b53d1bb4:::smithi19039307-22:head [read 0~360448,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:43.636 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.622306+0000", 2023-11-01T04:16:43.636 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 26.961412360000001, 2023-11-01T04:16:43.636 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.406118961, 2023-11-01T04:16:43.636 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.636 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:43.637 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:43.637 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:43.637 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:43.637 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 174 2023-11-01T04:16:43.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.637 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.638 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.638 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622306+0000", 2023-11-01T04:16:43.638 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.639 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.639 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622306+0000", 2023-11-01T04:16:43.639 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.640 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.640 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622307+0000", 2023-11-01T04:16:43.640 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6899999999999998e-07 2023-11-01T04:16:43.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.640 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.640 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.641 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622309+0000", 2023-11-01T04:16:43.641 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3580000000000001e-06 2023-11-01T04:16:43.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.641 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.641 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622315+0000", 2023-11-01T04:16:43.642 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3110000000000002e-06 2023-11-01T04:16:43.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.642 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.642 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.642 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.622321+0000", 2023-11-01T04:16:43.643 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8000000000000004e-06 2023-11-01T04:16:43.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.643 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.643 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.643 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623233+0000", 2023-11-01T04:16:43.643 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00091144800000000003 2023-11-01T04:16:43.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.644 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:43.644 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.623341+0000", 2023-11-01T04:16:43.644 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000108343 2023-11-01T04:16:43.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.645 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.645 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.645 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.028425+0000", 2023-11-01T04:16:43.645 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.40508433199999999 2023-11-01T04:16:43.645 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.645 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.646 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.646 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.646 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/31 ECSubReadReply(tid=39, attrs_read=0))", 2023-11-01T04:16:43.646 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.624361+0000", 2023-11-01T04:16:43.646 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 26.959357884999999, 2023-11-01T04:16:43.647 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.40370825399999999, 2023-11-01T04:16:43.647 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.647 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.647 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.647 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.647 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624361+0000", 2023-11-01T04:16:43.648 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.648 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624361+0000", 2023-11-01T04:16:43.649 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.649 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.649 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.649 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624361+0000", 2023-11-01T04:16:43.649 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.650 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.650 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.650 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.624361+0000", 2023-11-01T04:16:43.650 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.651 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.651 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.651 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.723932+0000", 2023-11-01T04:16:43.651 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.099571006000000004 2023-11-01T04:16:43.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.651 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.652 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.652 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.723938+0000", 2023-11-01T04:16:43.652 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3500000000000002e-06 2023-11-01T04:16:43.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.652 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.652 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.653 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.027693+0000", 2023-11-01T04:16:43.653 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.303755302 2023-11-01T04:16:43.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.653 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.653 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.653 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.028069+0000", 2023-11-01T04:16:43.654 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00037559600000000001 2023-11-01T04:16:43.654 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.654 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.654 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.654 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.655 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/31 ECSubReadReply(tid=40, attrs_read=0))", 2023-11-01T04:16:43.655 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.724781+0000", 2023-11-01T04:16:43.655 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 26.858937631, 2023-11-01T04:16:43.655 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30365672900000001, 2023-11-01T04:16:43.655 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.655 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.656 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.656 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.656 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.656 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724781+0000", 2023-11-01T04:16:43.656 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.657 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.657 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.657 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724781+0000", 2023-11-01T04:16:43.657 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.657 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.658 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.658 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724781+0000", 2023-11-01T04:16:43.658 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.658 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.658 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.659 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.659 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724781+0000", 2023-11-01T04:16:43.659 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.659 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.659 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.659 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.660 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724823+0000", 2023-11-01T04:16:43.660 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2429000000000003e-05 2023-11-01T04:16:43.660 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.660 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.660 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.660 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724844+0000", 2023-11-01T04:16:43.661 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0305999999999998e-05 2023-11-01T04:16:43.661 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.661 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.661 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.661 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.028176+0000", 2023-11-01T04:16:43.661 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30333263500000002 2023-11-01T04:16:43.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.662 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.662 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.662 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.028438+0000", 2023-11-01T04:16:43.662 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00026135900000000002 2023-11-01T04:16:43.663 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.663 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.663 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.663 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.ds0 36/31 ECSubReadReply(tid=40, attrs_read=0))", 2023-11-01T04:16:43.664 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.724986+0000", 2023-11-01T04:16:43.664 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 26.858732333999999, 2023-11-01T04:16:43.664 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.303171529, 2023-11-01T04:16:43.664 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.664 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.665 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.665 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.665 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.665 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724986+0000", 2023-11-01T04:16:43.665 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.665 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.666 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.666 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.666 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724986+0000", 2023-11-01T04:16:43.666 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.666 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.667 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.667 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.724987+0000", 2023-11-01T04:16:43.667 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.114e-06 2023-11-01T04:16:43.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.667 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.667 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.668 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.725128+0000", 2023-11-01T04:16:43.668 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000140362 2023-11-01T04:16:43.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.668 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.668 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.668 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.725146+0000", 2023-11-01T04:16:43.669 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8768999999999999e-05 2023-11-01T04:16:43.669 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.669 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.669 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.669 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.825238+0000", 2023-11-01T04:16:43.669 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100091351 2023-11-01T04:16:43.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.670 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.670 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.670 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.028123+0000", 2023-11-01T04:16:43.670 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.202885538 2023-11-01T04:16:43.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.671 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.671 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.671 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.028158+0000", 2023-11-01T04:16:43.671 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4394999999999998e-05 2023-11-01T04:16:43.671 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.671 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.672 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.672 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:189 3.ds0 3:ba00e0a8:::smithi19039307-23:head [copy-from ver 6] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:43.672 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.964914+0000", 2023-11-01T04:16:43.672 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 26.618804131000001, 2023-11-01T04:16:43.673 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.50623075299999998, 2023-11-01T04:16:43.673 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.673 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:43.673 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:43.673 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:43.673 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:43.674 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 189 2023-11-01T04:16:43.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.674 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.674 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.674 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.964914+0000", 2023-11-01T04:16:43.675 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.675 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.675 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.964914+0000", 2023-11-01T04:16:43.675 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.676 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.676 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.964917+0000", 2023-11-01T04:16:43.676 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2170000000000001e-06 2023-11-01T04:16:43.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.677 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.677 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.677 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.964929+0000", 2023-11-01T04:16:43.677 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2269000000000001e-05 2023-11-01T04:16:43.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.677 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.678 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.678 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.964967+0000", 2023-11-01T04:16:43.678 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.7778999999999999e-05 2023-11-01T04:16:43.678 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.678 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.678 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.679 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.065061+0000", 2023-11-01T04:16:43.679 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10009433700000001 2023-11-01T04:16:43.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.679 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.679 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.065209+0000", 2023-11-01T04:16:43.680 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014821100000000001 2023-11-01T04:16:43.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.680 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.680 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:43.680 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.065350+0000", 2023-11-01T04:16:43.681 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000140934 2023-11-01T04:16:43.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.681 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.681 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.681 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.266989+0000", 2023-11-01T04:16:43.681 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.201639125 2023-11-01T04:16:43.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.682 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.682 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.682 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.269850+0000", 2023-11-01T04:16:43.682 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00286075 2023-11-01T04:16:43.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.683 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.683 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:16:43.683 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.470971+0000", 2023-11-01T04:16:43.683 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.201120576 2023-11-01T04:16:43.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.683 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.684 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.684 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.471145+0000", 2023-11-01T04:16:43.684 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00017455500000000001 2023-11-01T04:16:43.684 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.684 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.684 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.685 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.685 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.685 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:266 3.ds0 3:bc424c1c:::smithi19039307-35:head [append 0~606208] snapc 5=[5,4,3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2023-11-01T04:16:43.685 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.010238+0000", 2023-11-01T04:16:43.685 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 22.573481059999999, 2023-11-01T04:16:43.685 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.41368528799999998, 2023-11-01T04:16:43.686 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.686 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:43.686 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:43.686 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:43.686 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:43.686 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 266 2023-11-01T04:16:43.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.687 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.687 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.687 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.687 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.010238+0000", 2023-11-01T04:16:43.688 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.688 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.688 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.688 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.688 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.010238+0000", 2023-11-01T04:16:43.689 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.689 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.689 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.689 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.689 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.010241+0000", 2023-11-01T04:16:43.689 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6899999999999998e-06 2023-11-01T04:16:43.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.690 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.690 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.690 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.011075+0000", 2023-11-01T04:16:43.690 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00083363100000000004 2023-11-01T04:16:43.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.691 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.691 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.691 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.011102+0000", 2023-11-01T04:16:43.691 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6883999999999999e-05 2023-11-01T04:16:43.691 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.692 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.692 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.692 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.011119+0000", 2023-11-01T04:16:43.692 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6915e-05 2023-11-01T04:16:43.692 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.692 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.693 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.693 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.011208+0000", 2023-11-01T04:16:43.693 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8889999999999998e-05 2023-11-01T04:16:43.693 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.693 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.693 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2023-11-01T04:16:43.694 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.011213+0000", 2023-11-01T04:16:43.694 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3000000000000001e-06 2023-11-01T04:16:43.694 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.694 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.694 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.694 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.117559+0000", 2023-11-01T04:16:43.695 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.106346042 2023-11-01T04:16:43.695 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.695 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.695 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:43.695 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.117672+0000", 2023-11-01T04:16:43.695 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011343100000000001 2023-11-01T04:16:43.696 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.696 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.696 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.696 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.118896+0000", 2023-11-01T04:16:43.696 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0012238329999999999 2023-11-01T04:16:43.696 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.697 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.697 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.697 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122513+0000", 2023-11-01T04:16:43.697 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0036173350000000002 2023-11-01T04:16:43.697 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.697 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.698 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:16:43.698 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.423805+0000", 2023-11-01T04:16:43.698 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30129125299999998 2023-11-01T04:16:43.698 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.698 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.698 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.699 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.423923+0000", 2023-11-01T04:16:43.699 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000118084 2023-11-01T04:16:43.699 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.699 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.699 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.699 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.700 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.700 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:267 3.ds0 3:bc424c1c:::smithi19039307-35:head [append 0~376832] snapc 5=[5,4,3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2023-11-01T04:16:43.700 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.016260+0000", 2023-11-01T04:16:43.700 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 22.567458299999998, 2023-11-01T04:16:43.700 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.40823882900000003, 2023-11-01T04:16:43.700 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.701 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:43.701 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:43.701 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:43.701 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:43.701 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 267 2023-11-01T04:16:43.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.702 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.702 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.702 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.702 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016260+0000", 2023-11-01T04:16:43.702 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.702 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.703 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.703 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.703 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016260+0000", 2023-11-01T04:16:43.703 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.703 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.703 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.704 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.704 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016262+0000", 2023-11-01T04:16:43.704 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.683e-06 2023-11-01T04:16:43.704 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.704 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.704 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.705 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016768+0000", 2023-11-01T04:16:43.705 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00050577 2023-11-01T04:16:43.705 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.705 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.705 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.705 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016794+0000", 2023-11-01T04:16:43.706 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6316e-05 2023-11-01T04:16:43.706 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.706 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.706 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.706 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016815+0000", 2023-11-01T04:16:43.707 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0959e-05 2023-11-01T04:16:43.707 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.707 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.707 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.707 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016986+0000", 2023-11-01T04:16:43.707 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00017064399999999999 2023-11-01T04:16:43.708 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.708 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.708 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:43.708 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016991+0000", 2023-11-01T04:16:43.708 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3539999999999999e-06 2023-11-01T04:16:43.708 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.708 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.709 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.709 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.119948+0000", 2023-11-01T04:16:43.709 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.102957328 2023-11-01T04:16:43.709 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.709 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.709 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:43.710 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.120043+0000", 2023-11-01T04:16:43.710 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4456999999999994e-05 2023-11-01T04:16:43.710 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.710 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.710 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.711 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.120770+0000", 2023-11-01T04:16:43.711 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00072736899999999995 2023-11-01T04:16:43.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.711 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.711 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.711 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123706+0000", 2023-11-01T04:16:43.711 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0029355520000000001 2023-11-01T04:16:43.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.712 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.712 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:16:43.712 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.424409+0000", 2023-11-01T04:16:43.712 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30070348299999999 2023-11-01T04:16:43.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.713 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.713 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.713 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.424499+0000", 2023-11-01T04:16:43.713 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9913999999999993e-05 2023-11-01T04:16:43.713 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.714 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.714 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.714 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.714 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.714 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:268 3.ds0 3:bc424c1c:::smithi19039307-35:head [setxattr _header (56) in=7b] snapc 5=[5,4,3] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2023-11-01T04:16:43.714 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.016855+0000", 2023-11-01T04:16:43.715 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 22.566863966, 2023-11-01T04:16:43.715 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.507789088, 2023-11-01T04:16:43.715 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.715 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:43.715 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:43.715 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:43.716 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:43.716 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 268 2023-11-01T04:16:43.716 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.716 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.716 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.717 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.717 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016855+0000", 2023-11-01T04:16:43.717 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.717 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.717 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.717 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.718 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016855+0000", 2023-11-01T04:16:43.718 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.718 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.718 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.718 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.718 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016856+0000", 2023-11-01T04:16:43.719 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.33e-06 2023-11-01T04:16:43.719 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.719 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.719 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.719 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016868+0000", 2023-11-01T04:16:43.719 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2306999999999999e-05 2023-11-01T04:16:43.720 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.720 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.720 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.720 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016892+0000", 2023-11-01T04:16:43.720 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3422e-05 2023-11-01T04:16:43.721 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.721 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.721 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.721 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016909+0000", 2023-11-01T04:16:43.721 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7793000000000001e-05 2023-11-01T04:16:43.721 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.721 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.722 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.722 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.017060+0000", 2023-11-01T04:16:43.722 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000150156 2023-11-01T04:16:43.722 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.722 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.723 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:43.723 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.017064+0000", 2023-11-01T04:16:43.723 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.98e-06 2023-11-01T04:16:43.723 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.723 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.723 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.724 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.121471+0000", 2023-11-01T04:16:43.724 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10440788500000001 2023-11-01T04:16:43.724 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.724 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.724 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:43.725 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.121586+0000", 2023-11-01T04:16:43.725 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000114838 2023-11-01T04:16:43.725 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.725 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.725 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.725 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122117+0000", 2023-11-01T04:16:43.726 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00053116399999999999 2023-11-01T04:16:43.726 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.726 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.726 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.726 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123808+0000", 2023-11-01T04:16:43.726 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0016909329999999999 2023-11-01T04:16:43.727 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.727 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.727 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:16:43.727 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.524498+0000", 2023-11-01T04:16:43.727 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.400689507 2023-11-01T04:16:43.727 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.728 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.728 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.728 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.524644+0000", 2023-11-01T04:16:43.728 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000145773 2023-11-01T04:16:43.728 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.728 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.729 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.729 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.729 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.729 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:269 3.ds0 3:bc424c1c:::smithi19039307-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2023-11-01T04:16:43.729 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.016953+0000", 2023-11-01T04:16:43.729 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 22.566765442000001, 2023-11-01T04:16:43.730 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.81260531499999999, 2023-11-01T04:16:43.730 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.730 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:43.730 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:43.730 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:43.730 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:43.731 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 269 2023-11-01T04:16:43.731 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.731 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.731 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.731 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.731 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016953+0000", 2023-11-01T04:16:43.732 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.732 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.732 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.732 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.732 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016953+0000", 2023-11-01T04:16:43.732 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.733 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.733 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.733 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.733 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016954+0000", 2023-11-01T04:16:43.733 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2330000000000001e-06 2023-11-01T04:16:43.733 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.734 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.734 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.734 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016975+0000", 2023-11-01T04:16:43.734 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0817000000000001e-05 2023-11-01T04:16:43.734 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.734 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.016997+0000", 2023-11-01T04:16:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1865000000000002e-05 2023-11-01T04:16:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.735 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.017013+0000", 2023-11-01T04:16:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6164e-05 2023-11-01T04:16:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.736 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.114130+0000", 2023-11-01T04:16:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.097116342999999994 2023-11-01T04:16:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.114139+0000", 2023-11-01T04:16:43.737 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.1360000000000003e-06 2023-11-01T04:16:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122397+0000", 2023-11-01T04:16:43.738 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0082583009999999991 2023-11-01T04:16:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for rw locks", 2023-11-01T04:16:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122499+0000", 2023-11-01T04:16:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000102407 2023-11-01T04:16:43.739 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.524752+0000", 2023-11-01T04:16:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.40225248200000002 2023-11-01T04:16:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.740 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.524905+0000", 2023-11-01T04:16:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015291600000000001 2023-11-01T04:16:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.741 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.829558+0000", 2023-11-01T04:16:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30465365100000003 2023-11-01T04:16:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.742 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 41/31 ECSubWriteReply(tid=56, last_complete=41'22, committed=1, applied=1))", 2023-11-01T04:16:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.124552+0000", 2023-11-01T04:16:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 22.459166891999999, 2023-11-01T04:16:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.29865597799999999, 2023-11-01T04:16:43.743 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.124552+0000", 2023-11-01T04:16:43.744 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.124552+0000", 2023-11-01T04:16:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.745 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.124553+0000", 2023-11-01T04:16:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.16e-07 2023-11-01T04:16:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.746 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.124559+0000", 2023-11-01T04:16:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3210000000000001e-06 2023-11-01T04:16:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.747 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.124578+0000", 2023-11-01T04:16:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9551e-05 2023-11-01T04:16:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.748 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.224689+0000", 2023-11-01T04:16:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100110328 2023-11-01T04:16:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.423157+0000", 2023-11-01T04:16:43.749 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19846841700000001 2023-11-01T04:16:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.423208+0000", 2023-11-01T04:16:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0544999999999998e-05 2023-11-01T04:16:43.750 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:352 3.ds0 3:b59be519:::smithi19039307-27:head [copy-from ver 22] snapc c=[c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:43.751 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.452180+0000", 2023-11-01T04:16:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 16.131538397, 2023-11-01T04:16:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.56919231299999995, 2023-11-01T04:16:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:43.752 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 352 2023-11-01T04:16:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.753 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.452180+0000", 2023-11-01T04:16:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.754 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.452180+0000", 2023-11-01T04:16:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.452184+0000", 2023-11-01T04:16:43.755 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.3830000000000001e-06 2023-11-01T04:16:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.452197+0000", 2023-11-01T04:16:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3890000000000001e-05 2023-11-01T04:16:43.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.452233+0000", 2023-11-01T04:16:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5111999999999999e-05 2023-11-01T04:16:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.757 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.758 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.758 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.452280+0000", 2023-11-01T04:16:43.758 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7063e-05 2023-11-01T04:16:43.758 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.758 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.758 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.759 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.452576+0000", 2023-11-01T04:16:43.759 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00029669599999999999 2023-11-01T04:16:43.759 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.759 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.759 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:43.759 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.452838+0000", 2023-11-01T04:16:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000262074 2023-11-01T04:16:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.820483+0000", 2023-11-01T04:16:43.760 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.36764486600000001 2023-11-01T04:16:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.827442+0000", 2023-11-01T04:16:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0069588899999999997 2023-11-01T04:16:43.761 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.762 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.762 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:16:43.762 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.021242+0000", 2023-11-01T04:16:43.762 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19380023800000001 2023-11-01T04:16:43.762 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.021373+0000", 2023-11-01T04:16:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013010100000000001 2023-11-01T04:16:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.763 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:353 3.ds0 3:b59be519:::smithi19039307-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.452341+0000", 2023-11-01T04:16:43.764 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 16.131377784000001, 2023-11-01T04:16:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.37340680500000001, 2023-11-01T04:16:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:43.765 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 353 2023-11-01T04:16:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.766 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.452341+0000", 2023-11-01T04:16:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.452341+0000", 2023-11-01T04:16:43.767 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.452342+0000", 2023-11-01T04:16:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.686e-06 2023-11-01T04:16:43.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.452349+0000", 2023-11-01T04:16:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4400000000000002e-06 2023-11-01T04:16:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.769 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.452370+0000", 2023-11-01T04:16:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1120000000000001e-05 2023-11-01T04:16:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.770 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.452391+0000", 2023-11-01T04:16:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1202000000000001e-05 2023-11-01T04:16:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.771 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.470829+0000", 2023-11-01T04:16:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.018438175000000001 2023-11-01T04:16:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for blocked object", 2023-11-01T04:16:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.470928+0000", 2023-11-01T04:16:43.772 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.8212000000000001e-05 2023-11-01T04:16:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.825208+0000", 2023-11-01T04:16:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.35428061300000002 2023-11-01T04:16:43.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.825480+0000", 2023-11-01T04:16:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000272 2023-11-01T04:16:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.774 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.825748+0000", 2023-11-01T04:16:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00026735700000000002 2023-11-01T04:16:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.775 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:362 3.as0 3:54622f20:::smithi19039307-19:head [copy-from ver 6] snapc c=[c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.511186+0000", 2023-11-01T04:16:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 16.072532687999999, 2023-11-01T04:16:43.776 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.61234525500000003, 2023-11-01T04:16:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:43.777 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 362 2023-11-01T04:16:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.511186+0000", 2023-11-01T04:16:43.778 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.511186+0000", 2023-11-01T04:16:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.511188+0000", 2023-11-01T04:16:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2670000000000001e-06 2023-11-01T04:16:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.780 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.511200+0000", 2023-11-01T04:16:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1693e-05 2023-11-01T04:16:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.781 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.511225+0000", 2023-11-01T04:16:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5551999999999999e-05 2023-11-01T04:16:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.782 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.511239+0000", 2023-11-01T04:16:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3454e-05 2023-11-01T04:16:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.783 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.611487+0000", 2023-11-01T04:16:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100248327 2023-11-01T04:16:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.611793+0000", 2023-11-01T04:16:43.784 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00030562599999999998 2023-11-01T04:16:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.918610+0000", 2023-11-01T04:16:43.785 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.306816741 2023-11-01T04:16:43.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.786 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.786 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.786 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.921812+0000", 2023-11-01T04:16:43.786 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0032021929999999999 2023-11-01T04:16:43.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.787 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.787 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:16:43.787 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.123428+0000", 2023-11-01T04:16:43.787 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20161631899999999 2023-11-01T04:16:43.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.787 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.788 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.788 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:28.123531+0000", 2023-11-01T04:16:43.788 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000103083 2023-11-01T04:16:43.788 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.789 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.789 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.789 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.789 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.789 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:363 3.as0 3:54622f20:::smithi19039307-19:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:43.789 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.511274+0000", 2023-11-01T04:16:43.790 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 16.072444253, 2023-11-01T04:16:43.790 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.40979745899999998, 2023-11-01T04:16:43.790 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.790 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:43.790 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:43.791 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:43.791 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:43.791 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 363 2023-11-01T04:16:43.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.791 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.792 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.792 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.511274+0000", 2023-11-01T04:16:43.792 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.793 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.793 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.511274+0000", 2023-11-01T04:16:43.793 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.793 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.793 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.794 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.511276+0000", 2023-11-01T04:16:43.794 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3629999999999999e-06 2023-11-01T04:16:43.794 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.794 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.794 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.794 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.511282+0000", 2023-11-01T04:16:43.795 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.387e-06 2023-11-01T04:16:43.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.795 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.795 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.795 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.511303+0000", 2023-11-01T04:16:43.795 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0888999999999999e-05 2023-11-01T04:16:43.796 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.796 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.796 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.796 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.611422+0000", 2023-11-01T04:16:43.796 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100119087 2023-11-01T04:16:43.796 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.797 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.797 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.797 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.612226+0000", 2023-11-01T04:16:43.797 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00080428100000000001 2023-11-01T04:16:43.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.797 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.798 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for blocked object", 2023-11-01T04:16:43.798 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.612328+0000", 2023-11-01T04:16:43.798 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010181 2023-11-01T04:16:43.798 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.798 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.798 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.799 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.920701+0000", 2023-11-01T04:16:43.799 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30837305399999998 2023-11-01T04:16:43.799 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.799 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.799 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:43.799 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.920852+0000", 2023-11-01T04:16:43.800 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000151213 2023-11-01T04:16:43.800 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.800 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.800 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.800 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.921072+0000", 2023-11-01T04:16:43.800 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00021937500000000001 2023-11-01T04:16:43.801 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.801 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.801 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.801 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.801 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.801 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(osd.12.22:6 3.ds0 3:b0d0a55f:::smithi19039307-8:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:43.802 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.612220+0000", 2023-11-01T04:16:43.802 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 15.971498753000001, 2023-11-01T04:16:43.802 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30361432599999999, 2023-11-01T04:16:43.802 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.802 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:43.803 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.803 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.803 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.803 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.612220+0000", 2023-11-01T04:16:43.803 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.803 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.804 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.804 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.804 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.612220+0000", 2023-11-01T04:16:43.804 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.804 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.805 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.805 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.612222+0000", 2023-11-01T04:16:43.805 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4710000000000001e-06 2023-11-01T04:16:43.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.805 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.805 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.806 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.612236+0000", 2023-11-01T04:16:43.806 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3913999999999999e-05 2023-11-01T04:16:43.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.806 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.806 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.806 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.612269+0000", 2023-11-01T04:16:43.807 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2861999999999998e-05 2023-11-01T04:16:43.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.807 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.807 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.807 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.712412+0000", 2023-11-01T04:16:43.807 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10014334 2023-11-01T04:16:43.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.808 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.808 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.808 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.712595+0000", 2023-11-01T04:16:43.808 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000183005 2023-11-01T04:16:43.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.809 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.809 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:43.809 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.712784+0000", 2023-11-01T04:16:43.809 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00018862600000000001 2023-11-01T04:16:43.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.809 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.810 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.810 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.915834+0000", 2023-11-01T04:16:43.810 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20305010800000001 2023-11-01T04:16:43.810 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.810 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.810 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.811 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.811 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.811 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:405 3.ds0 3:be91e0c6:::smithi19039307-28:head [create,setxattr h^blbcaNtSmmsQtZPdaRtPkhOOgN_Vi\\f`NtsZ[V (561) in=40b,setxattr T (17332) in=1b,setxattr UVoaZteYfmpthpcfl[gebaR\\TdqcNkhPRRdrgsX] (279) in=40b,setxattr bTPgfXb\\YXtXkQag\\Qh`hlXr[posd]]cje]^eebl (1121) in=40b,setxattr hm^gakWSOOOfNOTph^tma]iaNsaqfjqZYb\\seZNW (3297) in=40b,setxattr YiNn[]m_q`YlQlaV^mRn^ (17332) in=21b,setxattr \\tllpkXThmWXSkOopogrdtjY]f^a^sem_krafjfi (947) in=40b,setxattr RXiRZRQVSaUTaZmQT^bfqZhS^mZsqadaPdW]ijm` (1963) in=40b,setxattr WTmsTZhNpU`tZY\\^aROaNdmtYsjjsgWWboehPOP` (19508) in=40b,setxattr UZmWZeWaoiX]jmbn][UadnNkcZc_hdcjidtNsRYo (1999) in=40b,setxattr NSebjRYYQ`SbaatVjZWNab`cenkljRq`^]Xr\\jRm (11788) in=40b,setxattr ip`daUaop[rdbNPo\\^c[Wcelsl]a][RScQZ[kfa^ (18524) in=40b,setxattr qU_qcNcQpNkegs_Od`jieTXX[mto]lsR^jTcYprR (593) in=40b,setxattr dboVg`srq[bReagdR]fX\\Pe`Warrp]\\[XpSipTN[ (1683) in=40b,setxattr ZdTaVj[ScjsOiQ]a_]R[RQWoYohnjd`fi[dYmfs` (840) in=40b,setxattr N_[blmbpstWhtUgW]ZjZ[\\O_erqcnQN]R_V^U\\RT (1015) in=40b,setxattr jXbP_mNsadnq`UVfmc^lfZUs\\r_[ZnrahqTUk^Zp (383) in=40b,setxattr nmemd\\ipehj\\q^UVY_YtSrhZjcabNXhlai_noQSO (807) in=40b,setxattr Rbc\\cahhlcrckUioieUbQnrseNkZkdPf\\rUmNb]S (384) in=40b,setxattr ZiOkXqg^PobTdZgrtZ]]l^Rap_tfSQoUQqaefj]] (69) in=40b] snapc 12=[12,11,10,f,e,d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2023-11-01T04:16:43.811 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:33.106425+0000", 2023-11-01T04:16:43.811 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 10.47729373, 2023-11-01T04:16:43.811 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.31227879200000003, 2023-11-01T04:16:43.812 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:43.812 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:43.812 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:43.812 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:43.812 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:43.813 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 405 2023-11-01T04:16:43.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.813 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:43.813 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.813 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:43.813 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.106425+0000", 2023-11-01T04:16:43.814 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.814 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.814 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.814 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:43.814 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.106425+0000", 2023-11-01T04:16:43.815 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:43.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.815 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.815 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:43.815 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.106427+0000", 2023-11-01T04:16:43.815 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5229999999999998e-06 2023-11-01T04:16:43.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.816 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.816 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:43.816 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.106631+0000", 2023-11-01T04:16:43.816 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000203971 2023-11-01T04:16:43.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.817 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.817 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:43.817 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.106659+0000", 2023-11-01T04:16:43.817 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7710000000000001e-05 2023-11-01T04:16:43.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.817 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.818 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:43.818 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.206797+0000", 2023-11-01T04:16:43.818 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10013757399999999 2023-11-01T04:16:43.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.818 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.819 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.819 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.206925+0000", 2023-11-01T04:16:43.819 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012805600000000001 2023-11-01T04:16:43.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.819 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.819 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2023-11-01T04:16:43.820 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.206937+0000", 2023-11-01T04:16:43.820 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2576e-05 2023-11-01T04:16:43.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.820 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.820 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:43.820 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.310328+0000", 2023-11-01T04:16:43.820 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103390729 2023-11-01T04:16:43.821 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.821 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.821 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:43.821 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.310656+0000", 2023-11-01T04:16:43.821 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000327993 2023-11-01T04:16:43.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.822 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.822 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:43.822 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.313827+0000", 2023-11-01T04:16:43.822 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0031706099999999999 2023-11-01T04:16:43.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.823 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.823 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:43.823 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.319524+0000", 2023-11-01T04:16:43.823 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0056977460000000001 2023-11-01T04:16:43.823 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.823 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.824 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:16:43.824 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.418622+0000", 2023-11-01T04:16:43.824 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.099097925000000003 2023-11-01T04:16:43.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:43.824 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:43.824 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:43.825 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:33.418704+0000", 2023-11-01T04:16:43.825 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1378999999999995e-05 2023-11-01T04:16:43.825 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.825 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.825 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.825 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:43.826 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:43.826 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:43.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-01T04:16:43.827 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:43.826+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:43.928 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:43.925+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:43.958 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:43.958 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:43.958 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2023-11-01T04:16:43.959 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:43.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-01T04:16:44.029 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:44.027+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:44.130 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:44.129+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:44.137 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:44.138 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:44.138 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:16:44.138 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:16:44.138 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:44.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-01T04:16:44.207 INFO:tasks.rados.rados.0.smithi190.stdout:196: done (1 left) 2023-11-01T04:16:44.207 INFO:tasks.rados.rados.0.smithi190.stdout:201: done (0 left) 2023-11-01T04:16:44.208 INFO:tasks.rados.rados.0.smithi190.stdout:202: read oid 45 snap -1 2023-11-01T04:16:44.208 INFO:tasks.rados.rados.0.smithi190.stdout:202: expect (ObjNum 44 snap 0 seq_num 44) 2023-11-01T04:16:44.208 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 31 current snap is 19 2023-11-01T04:16:44.208 INFO:tasks.rados.rados.0.smithi190.stdout:203: seq_num 86 ranges {1343488=688128,2031616=540672,2572288=229376} 2023-11-01T04:16:44.231 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:44.228+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:44.233 INFO:tasks.rados.rados.0.smithi190.stdout:203: writing smithi19039307-31 from 1343488 to 2031616 tid 1 2023-11-01T04:16:44.240 INFO:tasks.rados.rados.0.smithi190.stdout:203: writing smithi19039307-31 from 2031616 to 2572288 tid 2 2023-11-01T04:16:44.243 INFO:tasks.rados.rados.0.smithi190.stdout:203: writing smithi19039307-31 from 2572288 to 2801664 tid 3 2023-11-01T04:16:44.243 INFO:tasks.rados.rados.0.smithi190.stdout:204: read oid 49 snap -1 2023-11-01T04:16:44.243 INFO:tasks.rados.rados.0.smithi190.stdout:204: expect (ObjNum 60 snap 5 seq_num 60) 2023-11-01T04:16:44.243 INFO:tasks.rados.rados.0.smithi190.stdout:205: read oid 38 snap -1 2023-11-01T04:16:44.244 INFO:tasks.rados.rados.0.smithi190.stdout:205: expect (ObjNum 37 snap 0 seq_num 37) 2023-11-01T04:16:44.244 INFO:tasks.rados.rados.0.smithi190.stdout:206: read oid 40 snap -1 2023-11-01T04:16:44.244 INFO:tasks.rados.rados.0.smithi190.stdout:206: expect (ObjNum 39 snap 0 seq_num 39) 2023-11-01T04:16:44.244 INFO:tasks.rados.rados.0.smithi190.stdout:207: snap_create 2023-11-01T04:16:44.321 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:44.322 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:16:44.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:16:44.322 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:16:44.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.322 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/31 ECSubWrite(tid=18, reqid=client.4559.0:125, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:44.323 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.281930+0000", 2023-11-01T04:16:44.323 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 29.037577126999999, 2023-11-01T04:16:44.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20154303100000001, 2023-11-01T04:16:44.323 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.324 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.324 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281930+0000", 2023-11-01T04:16:44.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281930+0000", 2023-11-01T04:16:44.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281931+0000", 2023-11-01T04:16:44.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6400000000000001e-07 2023-11-01T04:16:44.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281961+0000", 2023-11-01T04:16:44.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9621999999999999e-05 2023-11-01T04:16:44.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281987+0000", 2023-11-01T04:16:44.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6316e-05 2023-11-01T04:16:44.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382103+0000", 2023-11-01T04:16:44.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100116649 2023-11-01T04:16:44.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.482335+0000", 2023-11-01T04:16:44.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100231813 2023-11-01T04:16:44.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.482364+0000", 2023-11-01T04:16:44.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8829000000000001e-05 2023-11-01T04:16:44.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.483427+0000", 2023-11-01T04:16:44.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010627970000000001 2023-11-01T04:16:44.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.483473+0000", 2023-11-01T04:16:44.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6241e-05 2023-11-01T04:16:44.334 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.334 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.334 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.335 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/31 ECSubWrite(tid=4, reqid=client.4559.0:136, at_version=36'3, trim_to=0'0, roll_forward_to=33'2))", 2023-11-01T04:16:44.335 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.284706+0000", 2023-11-01T04:16:44.335 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 29.034801218999998, 2023-11-01T04:16:44.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202677317, 2023-11-01T04:16:44.336 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.336 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.336 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.284706+0000", 2023-11-01T04:16:44.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.284706+0000", 2023-11-01T04:16:44.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.284707+0000", 2023-11-01T04:16:44.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8599999999999998e-07 2023-11-01T04:16:44.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.285485+0000", 2023-11-01T04:16:44.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077885900000000002 2023-11-01T04:16:44.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.285511+0000", 2023-11-01T04:16:44.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5378999999999999e-05 2023-11-01T04:16:44.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385600+0000", 2023-11-01T04:16:44.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100089631 2023-11-01T04:16:44.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385749+0000", 2023-11-01T04:16:44.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014876399999999999 2023-11-01T04:16:44.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385771+0000", 2023-11-01T04:16:44.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1535e-05 2023-11-01T04:16:44.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.487315+0000", 2023-11-01T04:16:44.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101544735 2023-11-01T04:16:44.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.487383+0000", 2023-11-01T04:16:44.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7928000000000004e-05 2023-11-01T04:16:44.346 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.346 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.347 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.347 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 36/31 ECSubWrite(tid=14, reqid=client.4559.0:146, at_version=36'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:44.347 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.322569+0000", 2023-11-01T04:16:44.348 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.996938324999999, 2023-11-01T04:16:44.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20180657299999999, 2023-11-01T04:16:44.348 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.348 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.348 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322569+0000", 2023-11-01T04:16:44.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322569+0000", 2023-11-01T04:16:44.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322571+0000", 2023-11-01T04:16:44.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8029999999999999e-06 2023-11-01T04:16:44.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322759+0000", 2023-11-01T04:16:44.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018862300000000001 2023-11-01T04:16:44.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322790+0000", 2023-11-01T04:16:44.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0712e-05 2023-11-01T04:16:44.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322807+0000", 2023-11-01T04:16:44.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6566e-05 2023-11-01T04:16:44.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.423031+0000", 2023-11-01T04:16:44.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10022463600000001 2023-11-01T04:16:44.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.423046+0000", 2023-11-01T04:16:44.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4632e-05 2023-11-01T04:16:44.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524322+0000", 2023-11-01T04:16:44.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101275743 2023-11-01T04:16:44.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524376+0000", 2023-11-01T04:16:44.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3857999999999998e-05 2023-11-01T04:16:44.359 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.359 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.359 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.360 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 36/31 ECSubWrite(tid=16, reqid=client.4559.0:147, at_version=36'10, trim_to=0'0, roll_forward_to=36'8))", 2023-11-01T04:16:44.360 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.382654+0000", 2023-11-01T04:16:44.360 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.936853723999999, 2023-11-01T04:16:44.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14175697400000001, 2023-11-01T04:16:44.361 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.361 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.361 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382654+0000", 2023-11-01T04:16:44.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382654+0000", 2023-11-01T04:16:44.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382655+0000", 2023-11-01T04:16:44.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.933e-06 2023-11-01T04:16:44.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382823+0000", 2023-11-01T04:16:44.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000167806 2023-11-01T04:16:44.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.366 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382856+0000", 2023-11-01T04:16:44.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2255999999999997e-05 2023-11-01T04:16:44.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.366 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382926+0000", 2023-11-01T04:16:44.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0759999999999993e-05 2023-11-01T04:16:44.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.424403+0000", 2023-11-01T04:16:44.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.041477075000000002 2023-11-01T04:16:44.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.424416+0000", 2023-11-01T04:16:44.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2249000000000001e-05 2023-11-01T04:16:44.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524381+0000", 2023-11-01T04:16:44.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.099965719999999994 2023-11-01T04:16:44.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524411+0000", 2023-11-01T04:16:44.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9175000000000001e-05 2023-11-01T04:16:44.371 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.371 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.372 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.372 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 36/31 ECSubWrite(tid=17, reqid=client.4559.0:148, at_version=36'11, trim_to=0'0, roll_forward_to=36'8))", 2023-11-01T04:16:44.372 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.385040+0000", 2023-11-01T04:16:44.372 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.934467261000002, 2023-11-01T04:16:44.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14044638700000001, 2023-11-01T04:16:44.373 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.373 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.373 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385040+0000", 2023-11-01T04:16:44.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385040+0000", 2023-11-01T04:16:44.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.376 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385042+0000", 2023-11-01T04:16:44.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7129999999999999e-06 2023-11-01T04:16:44.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385087+0000", 2023-11-01T04:16:44.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4895000000000003e-05 2023-11-01T04:16:44.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385121+0000", 2023-11-01T04:16:44.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4180999999999997e-05 2023-11-01T04:16:44.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385145+0000", 2023-11-01T04:16:44.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3949000000000001e-05 2023-11-01T04:16:44.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.523994+0000", 2023-11-01T04:16:44.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13884963 2023-11-01T04:16:44.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524008+0000", 2023-11-01T04:16:44.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3719e-05 2023-11-01T04:16:44.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.525407+0000", 2023-11-01T04:16:44.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013987229999999999 2023-11-01T04:16:44.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.382 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.525486+0000", 2023-11-01T04:16:44.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9577000000000001e-05 2023-11-01T04:16:44.383 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.383 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.383 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.384 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 37/31 ECSubWrite(tid=22, reqid=client.4559.0:197, at_version=37'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:44.384 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:16:44.384 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.279235712999998, 2023-11-01T04:16:44.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.301677274, 2023-11-01T04:16:44.384 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.385 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.385 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:16:44.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:16:44.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.387 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.387 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.387 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040273+0000", 2023-11-01T04:16:44.387 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3370000000000001e-06 2023-11-01T04:16:44.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.388 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.388 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040307+0000", 2023-11-01T04:16:44.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3714999999999998e-05 2023-11-01T04:16:44.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.389 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.389 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.389 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040331+0000", 2023-11-01T04:16:44.389 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4409e-05 2023-11-01T04:16:44.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.389 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.390 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.390 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140479+0000", 2023-11-01T04:16:44.390 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10014764900000001 2023-11-01T04:16:44.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.391 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.391 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.391 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.240734+0000", 2023-11-01T04:16:44.391 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100255645 2023-11-01T04:16:44.391 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.392 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.392 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.392 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.240751+0000", 2023-11-01T04:16:44.392 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6852000000000001e-05 2023-11-01T04:16:44.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.393 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.393 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.393 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341894+0000", 2023-11-01T04:16:44.393 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101143302 2023-11-01T04:16:44.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.394 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.394 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.394 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341949+0000", 2023-11-01T04:16:44.394 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4364999999999999e-05 2023-11-01T04:16:44.395 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.395 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.395 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.395 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.395 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 37/31 ECSubWrite(tid=23, reqid=client.4559.0:198, at_version=37'10, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:44.396 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.140534+0000", 2023-11-01T04:16:44.396 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.178972818999998, 2023-11-01T04:16:44.396 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202392451, 2023-11-01T04:16:44.396 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.396 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.397 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.397 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.397 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140534+0000", 2023-11-01T04:16:44.397 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140534+0000", 2023-11-01T04:16:44.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140536+0000", 2023-11-01T04:16:44.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.22e-06 2023-11-01T04:16:44.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140578+0000", 2023-11-01T04:16:44.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2715e-05 2023-11-01T04:16:44.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140603+0000", 2023-11-01T04:16:44.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4609000000000001e-05 2023-11-01T04:16:44.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.402 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140616+0000", 2023-11-01T04:16:44.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2763e-05 2023-11-01T04:16:44.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.402 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.403 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341493+0000", 2023-11-01T04:16:44.403 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.200877214 2023-11-01T04:16:44.403 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.403 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.403 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.403 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341508+0000", 2023-11-01T04:16:44.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5013e-05 2023-11-01T04:16:44.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.404 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.404 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.342869+0000", 2023-11-01T04:16:44.405 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001361183 2023-11-01T04:16:44.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.405 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.405 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.342927+0000", 2023-11-01T04:16:44.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7734e-05 2023-11-01T04:16:44.406 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.406 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.406 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.407 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 37/31 ECSubWrite(tid=13, reqid=client.4559.0:181, at_version=37'8, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:44.407 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.248764+0000", 2023-11-01T04:16:44.407 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.070743434000001, 2023-11-01T04:16:44.407 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20321630399999999, 2023-11-01T04:16:44.407 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.408 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.408 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.408 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.248764+0000", 2023-11-01T04:16:44.409 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.409 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.409 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.409 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.409 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.248764+0000", 2023-11-01T04:16:44.410 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.410 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.410 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.248766+0000", 2023-11-01T04:16:44.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8300000000000001e-06 2023-11-01T04:16:44.411 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.411 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.411 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.411 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.249255+0000", 2023-11-01T04:16:44.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00048892399999999996 2023-11-01T04:16:44.412 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.412 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.412 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.412 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.249286+0000", 2023-11-01T04:16:44.412 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1004999999999999e-05 2023-11-01T04:16:44.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.249302+0000", 2023-11-01T04:16:44.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6169999999999999e-05 2023-11-01T04:16:44.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.414 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.414 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.349516+0000", 2023-11-01T04:16:44.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10021408599999999 2023-11-01T04:16:44.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.415 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.415 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.415 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.349530+0000", 2023-11-01T04:16:44.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4472e-05 2023-11-01T04:16:44.415 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.416 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.416 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451931+0000", 2023-11-01T04:16:44.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10240034100000001 2023-11-01T04:16:44.416 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451980+0000", 2023-11-01T04:16:44.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9475999999999999e-05 2023-11-01T04:16:44.417 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.417 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.418 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.418 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 37/31 ECSubWrite(tid=14, reqid=client.4559.0:182, at_version=37'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:44.418 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.253221+0000", 2023-11-01T04:16:44.419 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.066286278, 2023-11-01T04:16:44.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19911255999999999, 2023-11-01T04:16:44.419 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.419 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.419 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253221+0000", 2023-11-01T04:16:44.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253221+0000", 2023-11-01T04:16:44.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253222+0000", 2023-11-01T04:16:44.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1570000000000001e-06 2023-11-01T04:16:44.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.423 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253418+0000", 2023-11-01T04:16:44.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019597500000000001 2023-11-01T04:16:44.423 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.423 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.423 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.423 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253443+0000", 2023-11-01T04:16:44.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5055999999999999e-05 2023-11-01T04:16:44.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253459+0000", 2023-11-01T04:16:44.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5495999999999999e-05 2023-11-01T04:16:44.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451029+0000", 2023-11-01T04:16:44.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19757031799999999 2023-11-01T04:16:44.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451039+0000", 2023-11-01T04:16:44.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0062e-05 2023-11-01T04:16:44.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.427 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.427 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.452298+0000", 2023-11-01T04:16:44.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012584849999999999 2023-11-01T04:16:44.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.452334+0000", 2023-11-01T04:16:44.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6010999999999998e-05 2023-11-01T04:16:44.428 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.428 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.429 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.429 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.2s5 37/31 ECSubWrite(tid=206, reqid=client.4559.0:176, at_version=37'19, trim_to=0'0, roll_forward_to=36'16))", 2023-11-01T04:16:44.429 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.253930+0000", 2023-11-01T04:16:44.430 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.065577162, 2023-11-01T04:16:44.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10315031600000001, 2023-11-01T04:16:44.430 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.430 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.430 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.431 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.431 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253930+0000", 2023-11-01T04:16:44.431 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.431 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.431 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.432 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.432 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253930+0000", 2023-11-01T04:16:44.432 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.432 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.432 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253930+0000", 2023-11-01T04:16:44.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5999999999999999e-07 2023-11-01T04:16:44.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.254197+0000", 2023-11-01T04:16:44.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026679500000000001 2023-11-01T04:16:44.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.254222+0000", 2023-11-01T04:16:44.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4453000000000001e-05 2023-11-01T04:16:44.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.435 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.254237+0000", 2023-11-01T04:16:44.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4946000000000001e-05 2023-11-01T04:16:44.436 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.436 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.436 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.436 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.354451+0000", 2023-11-01T04:16:44.436 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10021408599999999 2023-11-01T04:16:44.436 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.437 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.437 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.437 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.354465+0000", 2023-11-01T04:16:44.437 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4549000000000001e-05 2023-11-01T04:16:44.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.437 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.438 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.438 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.357011+0000", 2023-11-01T04:16:44.438 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025457520000000001 2023-11-01T04:16:44.438 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.438 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.357080+0000", 2023-11-01T04:16:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9375000000000006e-05 2023-11-01T04:16:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.439 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.440 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.440 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 37/31 ECSubWrite(tid=15, reqid=client.4559.0:183, at_version=37'10, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:44.440 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.255859+0000", 2023-11-01T04:16:44.440 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.063648585999999, 2023-11-01T04:16:44.440 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19868693900000001, 2023-11-01T04:16:44.441 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.441 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.441 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.441 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255859+0000", 2023-11-01T04:16:44.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.442 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255859+0000", 2023-11-01T04:16:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255860+0000", 2023-11-01T04:16:44.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2300000000000001e-06 2023-11-01T04:16:44.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.444 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.444 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.444 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255894+0000", 2023-11-01T04:16:44.444 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3796e-05 2023-11-01T04:16:44.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.445 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.445 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.445 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255923+0000", 2023-11-01T04:16:44.445 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9352e-05 2023-11-01T04:16:44.445 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.445 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.446 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.446 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255936+0000", 2023-11-01T04:16:44.446 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3232e-05 2023-11-01T04:16:44.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.446 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.446 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451650+0000", 2023-11-01T04:16:44.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19571368 2023-11-01T04:16:44.447 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.447 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451665+0000", 2023-11-01T04:16:44.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5119e-05 2023-11-01T04:16:44.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.448 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.454503+0000", 2023-11-01T04:16:44.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028377400000000001 2023-11-01T04:16:44.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.454546+0000", 2023-11-01T04:16:44.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2790000000000002e-05 2023-11-01T04:16:44.449 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.450 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.450 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.450 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.450 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpRead(3.es2 41/31 ECSubRead(tid=31, to_read={3:706dddd5:::smithi19039307-14:head=0,159744,0}, subchunks={3:706dddd5:::smithi19039307-14:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:44.450 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:21.022133+0000", 2023-11-01T04:16:44.451 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 23.297374576999999, 2023-11-01T04:16:44.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16269023099999999, 2023-11-01T04:16:44.451 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.451 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.451 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022133+0000", 2023-11-01T04:16:44.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022133+0000", 2023-11-01T04:16:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022133+0000", 2023-11-01T04:16:44.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2e-07 2023-11-01T04:16:44.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022144+0000", 2023-11-01T04:16:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.118e-05 2023-11-01T04:16:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022175+0000", 2023-11-01T04:16:44.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0518999999999997e-05 2023-11-01T04:16:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022204+0000", 2023-11-01T04:16:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8878999999999999e-05 2023-11-01T04:16:44.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.184034+0000", 2023-11-01T04:16:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.161829903 2023-11-01T04:16:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.184823+0000", 2023-11-01T04:16:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00078932999999999998 2023-11-01T04:16:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.458 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpRead(3.es2 41/31 ECSubRead(tid=32, to_read={3:706dddd5:::smithi19039307-14:head=0,159744,0}, subchunks={3:706dddd5:::smithi19039307-14:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:21.023180+0000", 2023-11-01T04:16:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 23.296326910000001, 2023-11-01T04:16:44.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16201669099999999, 2023-11-01T04:16:44.460 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.460 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.460 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023180+0000", 2023-11-01T04:16:44.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023180+0000", 2023-11-01T04:16:44.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023182+0000", 2023-11-01T04:16:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0470000000000001e-06 2023-11-01T04:16:44.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023203+0000", 2023-11-01T04:16:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0845000000000001e-05 2023-11-01T04:16:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023244+0000", 2023-11-01T04:16:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1118999999999997e-05 2023-11-01T04:16:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023280+0000", 2023-11-01T04:16:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5812000000000002e-05 2023-11-01T04:16:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.184966+0000", 2023-11-01T04:16:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16168543899999999 2023-11-01T04:16:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.185197+0000", 2023-11-01T04:16:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023142900000000001 2023-11-01T04:16:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 41/31 ECSubWrite(tid=37, reqid=osd.11.0:36, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2023-11-01T04:16:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:21.644001+0000", 2023-11-01T04:16:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 22.675506424000002, 2023-11-01T04:16:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.103507961, 2023-11-01T04:16:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.468 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.644001+0000", 2023-11-01T04:16:44.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.644001+0000", 2023-11-01T04:16:44.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.644003+0000", 2023-11-01T04:16:44.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7969999999999999e-06 2023-11-01T04:16:44.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.644025+0000", 2023-11-01T04:16:44.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2566e-05 2023-11-01T04:16:44.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.644055+0000", 2023-11-01T04:16:44.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0199e-05 2023-11-01T04:16:44.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.644071+0000", 2023-11-01T04:16:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5925000000000001e-05 2023-11-01T04:16:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.644187+0000", 2023-11-01T04:16:44.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115736 2023-11-01T04:16:44.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.644201+0000", 2023-11-01T04:16:44.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4387e-05 2023-11-01T04:16:44.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.747416+0000", 2023-11-01T04:16:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.103214847 2023-11-01T04:16:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.747509+0000", 2023-11-01T04:16:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2503999999999994e-05 2023-11-01T04:16:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.477 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 48/31 ECSubWrite(tid=54, reqid=client.4559.0:369, at_version=48'22, trim_to=0'0, roll_forward_to=48'20))", 2023-11-01T04:16:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:28.168501+0000", 2023-11-01T04:16:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 16.151005973, 2023-11-01T04:16:44.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.198147297, 2023-11-01T04:16:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.168501+0000", 2023-11-01T04:16:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.168501+0000", 2023-11-01T04:16:44.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.168503+0000", 2023-11-01T04:16:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.55e-06 2023-11-01T04:16:44.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.168567+0000", 2023-11-01T04:16:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4234999999999997e-05 2023-11-01T04:16:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.168603+0000", 2023-11-01T04:16:44.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5536e-05 2023-11-01T04:16:44.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.168625+0000", 2023-11-01T04:16:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2671999999999999e-05 2023-11-01T04:16:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.365641+0000", 2023-11-01T04:16:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.197015577 2023-11-01T04:16:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.365647+0000", 2023-11-01T04:16:44.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3049999999999998e-06 2023-11-01T04:16:44.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.366616+0000", 2023-11-01T04:16:44.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00096908400000000005 2023-11-01T04:16:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.366649+0000", 2023-11-01T04:16:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2338000000000001e-05 2023-11-01T04:16:44.487 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.488 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.488 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.488 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 48/31 ECSubWrite(tid=55, reqid=client.4559.0:370, at_version=48'23, trim_to=0'0, roll_forward_to=48'20))", 2023-11-01T04:16:44.488 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:28.170472+0000", 2023-11-01T04:16:44.489 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 16.149035255000001, 2023-11-01T04:16:44.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19697836399999999, 2023-11-01T04:16:44.489 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.489 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.489 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.170472+0000", 2023-11-01T04:16:44.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.170472+0000", 2023-11-01T04:16:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.170473+0000", 2023-11-01T04:16:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1370000000000001e-06 2023-11-01T04:16:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.492 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.170502+0000", 2023-11-01T04:16:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8642000000000001e-05 2023-11-01T04:16:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.170525+0000", 2023-11-01T04:16:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3349999999999998e-05 2023-11-01T04:16:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.270635+0000", 2023-11-01T04:16:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100109552 2023-11-01T04:16:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.366031+0000", 2023-11-01T04:16:44.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.095395876000000004 2023-11-01T04:16:44.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.366038+0000", 2023-11-01T04:16:44.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0859999999999996e-06 2023-11-01T04:16:44.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.367415+0000", 2023-11-01T04:16:44.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013778429999999999 2023-11-01T04:16:44.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.367450+0000", 2023-11-01T04:16:44.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4878e-05 2023-11-01T04:16:44.498 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.498 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.499 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.499 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 58/31 ECSubWrite(tid=62, reqid=osd.7.0:62, at_version=58'55, trim_to=0'0, roll_forward_to=57'53))", 2023-11-01T04:16:44.499 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:38.346969+0000", 2023-11-01T04:16:44.499 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.9725384369999999, 2023-11-01T04:16:44.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104222194, 2023-11-01T04:16:44.500 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.500 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.500 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.346969+0000", 2023-11-01T04:16:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.346969+0000", 2023-11-01T04:16:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.346969+0000", 2023-11-01T04:16:44.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7599999999999997e-07 2023-11-01T04:16:44.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.347005+0000", 2023-11-01T04:16:44.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6028999999999997e-05 2023-11-01T04:16:44.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.347032+0000", 2023-11-01T04:16:44.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6996999999999999e-05 2023-11-01T04:16:44.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447129+0000", 2023-11-01T04:16:44.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100096649 2023-11-01T04:16:44.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447281+0000", 2023-11-01T04:16:44.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000152063 2023-11-01T04:16:44.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447306+0000", 2023-11-01T04:16:44.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4876e-05 2023-11-01T04:16:44.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.451098+0000", 2023-11-01T04:16:44.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037921230000000001 2023-11-01T04:16:44.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.451191+0000", 2023-11-01T04:16:44.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2980999999999996e-05 2023-11-01T04:16:44.509 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.509 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.509 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.510 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 58/31 ECSubWrite(tid=63, reqid=osd.7.0:63, at_version=58'57, trim_to=0'0, roll_forward_to=57'53))", 2023-11-01T04:16:44.510 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:38.447186+0000", 2023-11-01T04:16:44.510 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.8723213640000003, 2023-11-01T04:16:44.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202892298, 2023-11-01T04:16:44.510 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447186+0000", 2023-11-01T04:16:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447186+0000", 2023-11-01T04:16:44.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447187+0000", 2023-11-01T04:16:44.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.333e-06 2023-11-01T04:16:44.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447243+0000", 2023-11-01T04:16:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5461999999999999e-05 2023-11-01T04:16:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447268+0000", 2023-11-01T04:16:44.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5286000000000002e-05 2023-11-01T04:16:44.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447289+0000", 2023-11-01T04:16:44.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1060000000000002e-05 2023-11-01T04:16:44.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.647605+0000", 2023-11-01T04:16:44.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20031611899999999 2023-11-01T04:16:44.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.647619+0000", 2023-11-01T04:16:44.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4316e-05 2023-11-01T04:16:44.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.650003+0000", 2023-11-01T04:16:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002383994 2023-11-01T04:16:44.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.650078+0000", 2023-11-01T04:16:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4727999999999993e-05 2023-11-01T04:16:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.519 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 58/31 ECSubWrite(tid=64, reqid=osd.7.0:63, at_version=0'0, trim_to=0'0, roll_forward_to=58'57))", 2023-11-01T04:16:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:38.751255+0000", 2023-11-01T04:16:44.520 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.5682524960000004, 2023-11-01T04:16:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10327412, 2023-11-01T04:16:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.751255+0000", 2023-11-01T04:16:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.751255+0000", 2023-11-01T04:16:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.751256+0000", 2023-11-01T04:16:44.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.736e-06 2023-11-01T04:16:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.751279+0000", 2023-11-01T04:16:44.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2076e-05 2023-11-01T04:16:44.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.525 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.751308+0000", 2023-11-01T04:16:44.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9050000000000001e-05 2023-11-01T04:16:44.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.751324+0000", 2023-11-01T04:16:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6599000000000002e-05 2023-11-01T04:16:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.853207+0000", 2023-11-01T04:16:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101883101 2023-11-01T04:16:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.853226+0000", 2023-11-01T04:16:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8583000000000001e-05 2023-11-01T04:16:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.854453+0000", 2023-11-01T04:16:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001227283 2023-11-01T04:16:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.854529+0000", 2023-11-01T04:16:44.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5692000000000006e-05 2023-11-01T04:16:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.530 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.bs2 62/31 ECSubWrite(tid=70, reqid=client.4559.0:479, at_version=62'17, trim_to=0'0, roll_forward_to=45'15))", 2023-11-01T04:16:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:42.297514+0000", 2023-11-01T04:16:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.0219931299999998, 2023-11-01T04:16:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10530524099999999, 2023-11-01T04:16:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:44.531 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:44.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:44.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.297514+0000", 2023-11-01T04:16:44.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.297514+0000", 2023-11-01T04:16:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.297515+0000", 2023-11-01T04:16:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1300000000000001e-07 2023-11-01T04:16:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:44.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.298129+0000", 2023-11-01T04:16:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00061414599999999998 2023-11-01T04:16:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.298189+0000", 2023-11-01T04:16:44.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0109000000000003e-05 2023-11-01T04:16:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.298216+0000", 2023-11-01T04:16:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6903e-05 2023-11-01T04:16:44.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.398468+0000", 2023-11-01T04:16:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100251969 2023-11-01T04:16:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.398482+0000", 2023-11-01T04:16:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4749999999999999e-05 2023-11-01T04:16:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.402749+0000", 2023-11-01T04:16:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042666229999999998 2023-11-01T04:16:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.402819+0000", 2023-11-01T04:16:44.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0327999999999995e-05 2023-11-01T04:16:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:44.540 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:44.541 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:44.541 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-01T04:16:44.543 INFO:tasks.rados.rados.0.smithi190.stdout:205: expect (ObjNum 37 snap 0 seq_num 37) 2023-11-01T04:16:44.543 INFO:tasks.rados.rados.0.smithi190.stdout:206: expect (ObjNum 39 snap 0 seq_num 39) 2023-11-01T04:16:44.543 INFO:tasks.rados.rados.0.smithi190.stdout:203: finishing write tid 1 to smithi19039307-31 2023-11-01T04:16:44.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-01T04:16:44.544 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:44.541+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:44.555 INFO:tasks.rados.rados.0.smithi190.stdout:203: finishing write tid 2 to smithi19039307-31 2023-11-01T04:16:44.555 INFO:tasks.rados.rados.0.smithi190.stdout:203: finishing write tid 3 to smithi19039307-31 2023-11-01T04:16:44.615 INFO:tasks.rados.rados.0.smithi190.stdout:202: expect (ObjNum 44 snap 0 seq_num 44) 2023-11-01T04:16:44.643 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:44.638+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:44.696 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:44.696 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-11-01T04:16:44.716 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:44.716 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:44.717 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2023-11-01T04:16:44.717 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:44.717 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:44.718 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-01T04:16:44.723 INFO:tasks.rados.rados.0.smithi190.stdout:204: expect (ObjNum 60 snap 5 seq_num 60) 2023-11-01T04:16:44.731 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-01T04:16:44.738 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:44.738 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-01T04:16:44.744 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:44.744+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:44.759 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:44.760 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-01T04:16:44.778 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:44.778 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-01T04:16:44.794 INFO:tasks.rados.rados.0.smithi190.stdout:203: finishing write tid 4 to smithi19039307-31 2023-11-01T04:16:44.794 INFO:tasks.rados.rados.0.smithi190.stdout:202: done (5 left) 2023-11-01T04:16:44.794 INFO:tasks.rados.rados.0.smithi190.stdout:204: done (4 left) 2023-11-01T04:16:44.794 INFO:tasks.rados.rados.0.smithi190.stdout:205: done (3 left) 2023-11-01T04:16:44.795 INFO:tasks.rados.rados.0.smithi190.stdout:206: done (2 left) 2023-11-01T04:16:44.795 INFO:tasks.rados.rados.0.smithi190.stdout:207: done (1 left) 2023-11-01T04:16:44.795 INFO:tasks.rados.rados.0.smithi190.stdout:208: snap_remove snap 18 2023-11-01T04:16:44.796 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:44.796 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-01T04:16:44.816 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:44.817 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-01T04:16:44.836 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:44.837 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-01T04:16:44.845 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:44.846+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:44.853 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:44.853 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-01T04:16:44.854 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:44.854 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:44.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-01T04:16:44.873 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:44.873 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-01T04:16:44.894 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:44.894 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-01T04:16:44.915 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:44.915 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-01T04:16:44.932 INFO:teuthology.orchestra.run.smithi153.stdout:osd.12: {} 2023-11-01T04:16:44.932 INFO:teuthology.orchestra.run.smithi153.stderr:osd.12: osd_enable_op_tracker = 'false' 2023-11-01T04:16:44.946 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:44.944+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:44.951 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:44.952 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-01T04:16:44.974 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:44.974 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-01T04:16:44.991 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:44.991 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:44.995 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:44.996 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-01T04:16:45.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-01T04:16:45.049 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:45.049+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:45.124 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:45.124 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:45.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-01T04:16:45.148 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:45.147+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:45.214 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-11-01T04:16:45.260 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:45.260+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:45.304 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:45.304 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:45.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-01T04:16:45.365 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:45.365+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:45.435 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:45.436 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:45.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-01T04:16:45.451 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:45.446+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:45.551 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:45.547+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:45.618 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:45.618 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:45.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-01T04:16:45.652 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:45.650+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:45.752 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:45.751+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:45.755 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:45.755 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:45.764 INFO:teuthology.orchestra.run.smithi153.stderr:nodeep-scrub is set 2023-11-01T04:16:45.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-01T04:16:45.773 INFO:tasks.rados.rados.0.smithi190.stdout:208: done (1 left) 2023-11-01T04:16:45.773 INFO:tasks.rados.rados.0.smithi190.stdout:209: read oid 6 snap -1 2023-11-01T04:16:45.773 INFO:tasks.rados.rados.0.smithi190.stdout:209: expect (ObjNum 5 snap 0 seq_num 5) 2023-11-01T04:16:45.773 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 37 current snap is 20 2023-11-01T04:16:45.773 INFO:tasks.rados.rados.0.smithi190.stdout:203: finishing write tid 5 to smithi19039307-31 2023-11-01T04:16:45.774 INFO:tasks.rados.rados.0.smithi190.stdout:203: oid 31 updating version 0 to 41 2023-11-01T04:16:45.774 INFO:tasks.rados.rados.0.smithi190.stdout:203: oid 31 updating version 41 to 43 2023-11-01T04:16:45.774 INFO:tasks.rados.rados.0.smithi190.stdout:203: oid 31 version 43 is already newer than 40 2023-11-01T04:16:45.774 INFO:tasks.rados.rados.0.smithi190.stdout:203: oid 31 version 43 is already newer than 42 2023-11-01T04:16:45.775 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 31 v 43 (ObjNum 86 snap 19 seq_num 86) dirty exists 2023-11-01T04:16:45.775 INFO:tasks.rados.rados.0.smithi190.stdout:203: left oid 31 (ObjNum 86 snap 19 seq_num 86) 2023-11-01T04:16:45.775 INFO:tasks.rados.rados.0.smithi190.stdout:210: seq_num 87 ranges {606208=704512,1310720=688128,1998848=81920} 2023-11-01T04:16:45.794 INFO:tasks.rados.rados.0.smithi190.stdout:210: writing smithi19039307-37 from 606208 to 1310720 tid 1 2023-11-01T04:16:45.806 INFO:tasks.rados.rados.0.smithi190.stdout:210: writing smithi19039307-37 from 1310720 to 1998848 tid 2 2023-11-01T04:16:45.808 INFO:tasks.rados.rados.0.smithi190.stdout:210: writing smithi19039307-37 from 1998848 to 2080768 tid 3 2023-11-01T04:16:45.808 INFO:tasks.rados.rados.0.smithi190.stdout:203: done (2 left) 2023-11-01T04:16:45.809 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 9 current snap is 20 2023-11-01T04:16:45.809 INFO:tasks.rados.rados.0.smithi190.stdout:211: seq_num 88 ranges {147456=720896,868352=442368,1310720=327680} 2023-11-01T04:16:45.822 INFO:tasks.rados.rados.0.smithi190.stdout:211: writing smithi19039307-9 from 147456 to 868352 tid 1 2023-11-01T04:16:45.828 INFO:tasks.rados.rados.0.smithi190.stdout:211: writing smithi19039307-9 from 868352 to 1310720 tid 2 2023-11-01T04:16:45.833 INFO:tasks.rados.rados.0.smithi190.stdout:211: writing smithi19039307-9 from 1310720 to 1638400 tid 3 2023-11-01T04:16:45.834 INFO:tasks.rados.rados.0.smithi190.stdout:212: rmattr oid 10 current snap is 20 2023-11-01T04:16:45.834 INFO:tasks.rados.rados.0.smithi190.stdout:212: done (3 left) 2023-11-01T04:16:45.834 INFO:tasks.rados.rados.0.smithi190.stdout:213: snap_remove snap 5 2023-11-01T04:16:45.852 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:45.849+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:45.892 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:45.892 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:45.906 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-01T04:16:45.954 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:45.952+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:46.038 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:46.039 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:46.052 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-01T04:16:46.054 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:46.051+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:46.154 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:46.151+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:46.175 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:46.175 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:46.190 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-01T04:16:46.255 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:16:46.254+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:16:46.274 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2023-11-01T04:16:46.274 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-01T04:16:46.274 INFO:tasks.thrashosds.thrasher:Reweighting osd 5 to 0.8079104980447432 2023-11-01T04:16:46.275 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 0.8079104980447432 2023-11-01T04:16:46.356 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:46.354+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:46.356 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:46.356 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:46.370 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-01T04:16:46.457 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:46.455+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:46.493 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:46.493 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:46.507 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-01T04:16:46.558 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:46.554+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:46.658 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:46.655+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:46.670 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:46.670 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:46.685 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-01T04:16:46.759 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:46.756+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:46.768 INFO:teuthology.orchestra.run.smithi153.stderr:reweighted osd.5 to 0.80791 (ced3) 2023-11-01T04:16:46.774 INFO:tasks.rados.rados.0.smithi190.stdout:213: done (3 left) 2023-11-01T04:16:46.774 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 19 current snap is 20 2023-11-01T04:16:46.775 INFO:tasks.rados.rados.0.smithi190.stdout:214: seq_num 89 ranges {294912=737280,1032192=573440,1605632=196608} 2023-11-01T04:16:46.785 INFO:tasks.rados.rados.0.smithi190.stdout:214: writing smithi19039307-19 from 294912 to 1032192 tid 1 2023-11-01T04:16:46.792 INFO:tasks.rados.rados.0.smithi190.stdout:214: writing smithi19039307-19 from 1032192 to 1605632 tid 2 2023-11-01T04:16:46.794 INFO:tasks.rados.rados.0.smithi190.stdout:214: writing smithi19039307-19 from 1605632 to 1802240 tid 3 2023-11-01T04:16:46.794 INFO:tasks.rados.rados.0.smithi190.stdout:215: copy_from oid 36 from oid 45 current snap is 20 2023-11-01T04:16:46.795 INFO:tasks.rados.rados.0.smithi190.stdout:211: finishing write tid 1 to smithi19039307-9 2023-11-01T04:16:46.795 INFO:tasks.rados.rados.0.smithi190.stdout:210: finishing write tid 1 to smithi19039307-37 2023-11-01T04:16:46.795 INFO:tasks.rados.rados.0.smithi190.stdout:210: finishing write tid 2 to smithi19039307-37 2023-11-01T04:16:46.795 INFO:tasks.rados.rados.0.smithi190.stdout:209: expect (ObjNum 5 snap 0 seq_num 5) 2023-11-01T04:16:46.802 INFO:tasks.rados.rados.0.smithi190.stdout:210: finishing write tid 3 to smithi19039307-37 2023-11-01T04:16:46.802 INFO:tasks.rados.rados.0.smithi190.stdout:211: finishing write tid 2 to smithi19039307-9 2023-11-01T04:16:46.802 INFO:tasks.rados.rados.0.smithi190.stdout:210: finishing write tid 4 to smithi19039307-37 2023-11-01T04:16:46.802 INFO:tasks.rados.rados.0.smithi190.stdout:211: finishing write tid 3 to smithi19039307-9 2023-11-01T04:16:46.803 INFO:tasks.rados.rados.0.smithi190.stdout:209: done (4 left) 2023-11-01T04:16:46.803 INFO:tasks.rados.rados.0.smithi190.stdout:216: snap_remove snap 13 2023-11-01T04:16:46.851 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:46.852 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:46.859 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:16:46.858+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:16:46.865 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-01T04:16:46.960 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:46.958+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:46.992 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:46.993 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:47.007 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-01T04:16:47.014 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-01T04:16:47.060 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:47.059+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:47.126 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:47.126 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:47.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-01T04:16:47.161 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:47.161+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:47.171 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:47.171 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-11-01T04:16:47.193 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:47.193 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-01T04:16:47.211 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:47.211 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-01T04:16:47.232 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:47.232 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-01T04:16:47.250 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:47.251 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-01T04:16:47.262 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:47.259+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:47.271 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:47.271 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-01T04:16:47.291 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:47.291 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-01T04:16:47.313 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:47.313 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-01T04:16:47.314 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:47.314 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:47.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-01T04:16:47.331 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:47.331 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-01T04:16:47.352 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:47.352 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-01T04:16:47.363 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:16:47.358+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:16:47.371 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:47.372 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-01T04:16:47.392 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:47.392 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-01T04:16:47.394 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:47.392+0000 7fb1a78cf640 -1 snap_mapper.get_next_objects_to_trimNew Clone-Objects were added to Snap b after trimming was started 2023-11-01T04:16:47.395 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:47.392+0000 7fb1a78cf640 -1 snap_mapper.0x7fb1a78cb750 2023-11-01T04:16:47.399 INFO:tasks.ceph.osd.12.smithi153.stderr:./src/osd/SnapMapper.cc: In function 'int SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector*)' thread 7fb1a78cf640 time 2023-11-01T04:16:47.395771+0000 2023-11-01T04:16:47.400 INFO:tasks.ceph.osd.12.smithi153.stderr:./src/osd/SnapMapper.cc: 650: ceph_abort_msg("Clone-Objects were added to a trimmed snap") 2023-11-01T04:16:47.400 INFO:tasks.ceph.osd.12.smithi153.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:47.400 INFO:tasks.ceph.osd.12.smithi153.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x560de29fd9dc] 2023-11-01T04:16:47.400 INFO:tasks.ceph.osd.12.smithi153.stderr: 2: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x560de2ce1c9d] 2023-11-01T04:16:47.400 INFO:tasks.ceph.osd.12.smithi153.stderr: 3: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x560de2c4c2c2] 2023-11-01T04:16:47.401 INFO:tasks.ceph.osd.12.smithi153.stderr: 4: ceph-osd(+0xd7e721) [0x560de2aa7721] 2023-11-01T04:16:47.401 INFO:tasks.ceph.osd.12.smithi153.stderr: 5: ceph-osd(+0x1695cbb) [0x560de33becbb] 2023-11-01T04:16:47.401 INFO:tasks.ceph.osd.12.smithi153.stderr: 6: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x560de2be9288] 2023-11-01T04:16:47.401 INFO:tasks.ceph.osd.12.smithi153.stderr: 7: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x560de2d1f9df] 2023-11-01T04:16:47.401 INFO:tasks.ceph.osd.12.smithi153.stderr: 8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x560de2b3dd23] 2023-11-01T04:16:47.401 INFO:tasks.ceph.osd.12.smithi153.stderr: 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x560de2fef053] 2023-11-01T04:16:47.402 INFO:tasks.ceph.osd.12.smithi153.stderr: 10: ceph-osd(+0x12c6584) [0x560de2fef584] 2023-11-01T04:16:47.402 INFO:tasks.ceph.osd.12.smithi153.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb1c473eb43] 2023-11-01T04:16:47.402 INFO:tasks.ceph.osd.12.smithi153.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb1c47d0a00] 2023-11-01T04:16:47.402 INFO:tasks.ceph.osd.12.smithi153.stderr:*** Caught signal (Aborted) ** 2023-11-01T04:16:47.402 INFO:tasks.ceph.osd.12.smithi153.stderr: in thread 7fb1a78cf640 thread_name:tp_osd_tp 2023-11-01T04:16:47.403 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:47.396+0000 7fb1a78cf640 -1 ./src/osd/SnapMapper.cc: In function 'int SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector*)' thread 7fb1a78cf640 time 2023-11-01T04:16:47.395771+0000 2023-11-01T04:16:47.403 INFO:tasks.ceph.osd.12.smithi153.stderr:./src/osd/SnapMapper.cc: 650: ceph_abort_msg("Clone-Objects were added to a trimmed snap") 2023-11-01T04:16:47.403 INFO:tasks.ceph.osd.12.smithi153.stderr: 2023-11-01T04:16:47.403 INFO:tasks.ceph.osd.12.smithi153.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:47.403 INFO:tasks.ceph.osd.12.smithi153.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x560de29fd9dc] 2023-11-01T04:16:47.404 INFO:tasks.ceph.osd.12.smithi153.stderr: 2: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x560de2ce1c9d] 2023-11-01T04:16:47.404 INFO:tasks.ceph.osd.12.smithi153.stderr: 3: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x560de2c4c2c2] 2023-11-01T04:16:47.404 INFO:tasks.ceph.osd.12.smithi153.stderr: 4: ceph-osd(+0xd7e721) [0x560de2aa7721] 2023-11-01T04:16:47.404 INFO:tasks.ceph.osd.12.smithi153.stderr: 5: ceph-osd(+0x1695cbb) [0x560de33becbb] 2023-11-01T04:16:47.404 INFO:tasks.ceph.osd.12.smithi153.stderr: 6: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x560de2be9288] 2023-11-01T04:16:47.404 INFO:tasks.ceph.osd.12.smithi153.stderr: 7: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x560de2d1f9df] 2023-11-01T04:16:47.405 INFO:tasks.ceph.osd.12.smithi153.stderr: 8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x560de2b3dd23] 2023-11-01T04:16:47.405 INFO:tasks.ceph.osd.12.smithi153.stderr: 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x560de2fef053] 2023-11-01T04:16:47.405 INFO:tasks.ceph.osd.12.smithi153.stderr: 10: ceph-osd(+0x12c6584) [0x560de2fef584] 2023-11-01T04:16:47.405 INFO:tasks.ceph.osd.12.smithi153.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb1c473eb43] 2023-11-01T04:16:47.405 INFO:tasks.ceph.osd.12.smithi153.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb1c47d0a00] 2023-11-01T04:16:47.406 INFO:tasks.ceph.osd.12.smithi153.stderr: 2023-11-01T04:16:47.406 INFO:tasks.ceph.osd.12.smithi153.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:47.406 INFO:tasks.ceph.osd.12.smithi153.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb1c46ec520] 2023-11-01T04:16:47.406 INFO:tasks.ceph.osd.12.smithi153.stderr: 2: pthread_kill() 2023-11-01T04:16:47.407 INFO:tasks.ceph.osd.12.smithi153.stderr: 3: raise() 2023-11-01T04:16:47.407 INFO:tasks.ceph.osd.12.smithi153.stderr: 4: abort() 2023-11-01T04:16:47.407 INFO:tasks.ceph.osd.12.smithi153.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x560de29fda91] 2023-11-01T04:16:47.407 INFO:tasks.ceph.osd.12.smithi153.stderr: 6: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x560de2ce1c9d] 2023-11-01T04:16:47.407 INFO:tasks.ceph.osd.12.smithi153.stderr: 7: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x560de2c4c2c2] 2023-11-01T04:16:47.408 INFO:tasks.ceph.osd.12.smithi153.stderr: 8: ceph-osd(+0xd7e721) [0x560de2aa7721] 2023-11-01T04:16:47.408 INFO:tasks.ceph.osd.12.smithi153.stderr: 9: ceph-osd(+0x1695cbb) [0x560de33becbb] 2023-11-01T04:16:47.408 INFO:tasks.ceph.osd.12.smithi153.stderr: 10: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x560de2be9288] 2023-11-01T04:16:47.408 INFO:tasks.ceph.osd.12.smithi153.stderr: 11: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x560de2d1f9df] 2023-11-01T04:16:47.408 INFO:tasks.ceph.osd.12.smithi153.stderr: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x560de2b3dd23] 2023-11-01T04:16:47.409 INFO:tasks.ceph.osd.12.smithi153.stderr: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x560de2fef053] 2023-11-01T04:16:47.409 INFO:tasks.ceph.osd.12.smithi153.stderr: 14: ceph-osd(+0x12c6584) [0x560de2fef584] 2023-11-01T04:16:47.409 INFO:tasks.ceph.osd.12.smithi153.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb1c473eb43] 2023-11-01T04:16:47.409 INFO:tasks.ceph.osd.12.smithi153.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb1c47d0a00] 2023-11-01T04:16:47.409 INFO:tasks.ceph.osd.12.smithi153.stderr:2023-11-01T04:16:47.404+0000 7fb1a78cf640 -1 *** Caught signal (Aborted) ** 2023-11-01T04:16:47.410 INFO:tasks.ceph.osd.12.smithi153.stderr: in thread 7fb1a78cf640 thread_name:tp_osd_tp 2023-11-01T04:16:47.410 INFO:tasks.ceph.osd.12.smithi153.stderr: 2023-11-01T04:16:47.410 INFO:tasks.ceph.osd.12.smithi153.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:47.410 INFO:tasks.ceph.osd.12.smithi153.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb1c46ec520] 2023-11-01T04:16:47.410 INFO:tasks.ceph.osd.12.smithi153.stderr: 2: pthread_kill() 2023-11-01T04:16:47.411 INFO:tasks.ceph.osd.12.smithi153.stderr: 3: raise() 2023-11-01T04:16:47.411 INFO:tasks.ceph.osd.12.smithi153.stderr: 4: abort() 2023-11-01T04:16:47.411 INFO:tasks.ceph.osd.12.smithi153.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x560de29fda91] 2023-11-01T04:16:47.411 INFO:tasks.ceph.osd.12.smithi153.stderr: 6: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x560de2ce1c9d] 2023-11-01T04:16:47.411 INFO:tasks.ceph.osd.12.smithi153.stderr: 7: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x560de2c4c2c2] 2023-11-01T04:16:47.411 INFO:tasks.ceph.osd.12.smithi153.stderr: 8: ceph-osd(+0xd7e721) [0x560de2aa7721] 2023-11-01T04:16:47.412 INFO:tasks.ceph.osd.12.smithi153.stderr: 9: ceph-osd(+0x1695cbb) [0x560de33becbb] 2023-11-01T04:16:47.412 INFO:tasks.ceph.osd.12.smithi153.stderr: 10: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x560de2be9288] 2023-11-01T04:16:47.412 INFO:tasks.ceph.osd.12.smithi153.stderr: 11: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x560de2d1f9df] 2023-11-01T04:16:47.412 INFO:tasks.ceph.osd.12.smithi153.stderr: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x560de2b3dd23] 2023-11-01T04:16:47.412 INFO:tasks.ceph.osd.12.smithi153.stderr: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x560de2fef053] 2023-11-01T04:16:47.412 INFO:tasks.ceph.osd.12.smithi153.stderr: 14: ceph-osd(+0x12c6584) [0x560de2fef584] 2023-11-01T04:16:47.413 INFO:tasks.ceph.osd.12.smithi153.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb1c473eb43] 2023-11-01T04:16:47.413 INFO:tasks.ceph.osd.12.smithi153.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb1c47d0a00] 2023-11-01T04:16:47.413 INFO:tasks.ceph.osd.12.smithi153.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-01T04:16:47.413 INFO:tasks.ceph.osd.12.smithi153.stderr: 2023-11-01T04:16:47.414 INFO:tasks.ceph.osd.12.smithi153.stderr: -6922> 2023-11-01T04:16:44.744+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:47.414 INFO:tasks.ceph.osd.12.smithi153.stderr: -5906> 2023-11-01T04:16:45.260+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:47.418 INFO:tasks.ceph.osd.12.smithi153.stderr: -53> 2023-11-01T04:16:47.392+0000 7fb1a78cf640 -1 snap_mapper.get_next_objects_to_trimNew Clone-Objects were added to Snap b after trimming was started 2023-11-01T04:16:47.419 INFO:tasks.ceph.osd.12.smithi153.stderr: -52> 2023-11-01T04:16:47.392+0000 7fb1a78cf640 -1 snap_mapper.0x7fb1a78cb750 2023-11-01T04:16:47.419 INFO:tasks.ceph.osd.12.smithi153.stderr: -2> 2023-11-01T04:16:47.396+0000 7fb1a78cf640 -1 ./src/osd/SnapMapper.cc: In function 'int SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector*)' thread 7fb1a78cf640 time 2023-11-01T04:16:47.395771+0000 2023-11-01T04:16:47.419 INFO:tasks.ceph.osd.12.smithi153.stderr:./src/osd/SnapMapper.cc: 650: ceph_abort_msg("Clone-Objects were added to a trimmed snap") 2023-11-01T04:16:47.419 INFO:tasks.ceph.osd.12.smithi153.stderr: 2023-11-01T04:16:47.419 INFO:tasks.ceph.osd.12.smithi153.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:47.420 INFO:tasks.ceph.osd.12.smithi153.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x560de29fd9dc] 2023-11-01T04:16:47.420 INFO:tasks.ceph.osd.12.smithi153.stderr: 2: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x560de2ce1c9d] 2023-11-01T04:16:47.420 INFO:tasks.ceph.osd.12.smithi153.stderr: 3: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x560de2c4c2c2] 2023-11-01T04:16:47.420 INFO:tasks.ceph.osd.12.smithi153.stderr: 4: ceph-osd(+0xd7e721) [0x560de2aa7721] 2023-11-01T04:16:47.420 INFO:tasks.ceph.osd.12.smithi153.stderr: 5: ceph-osd(+0x1695cbb) [0x560de33becbb] 2023-11-01T04:16:47.420 INFO:tasks.ceph.osd.12.smithi153.stderr: 6: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x560de2be9288] 2023-11-01T04:16:47.421 INFO:tasks.ceph.osd.12.smithi153.stderr: 7: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x560de2d1f9df] 2023-11-01T04:16:47.421 INFO:tasks.ceph.osd.12.smithi153.stderr: 8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x560de2b3dd23] 2023-11-01T04:16:47.421 INFO:tasks.ceph.osd.12.smithi153.stderr: 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x560de2fef053] 2023-11-01T04:16:47.421 INFO:tasks.ceph.osd.12.smithi153.stderr: 10: ceph-osd(+0x12c6584) [0x560de2fef584] 2023-11-01T04:16:47.421 INFO:tasks.ceph.osd.12.smithi153.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb1c473eb43] 2023-11-01T04:16:47.421 INFO:tasks.ceph.osd.12.smithi153.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb1c47d0a00] 2023-11-01T04:16:47.422 INFO:tasks.ceph.osd.12.smithi153.stderr: 2023-11-01T04:16:47.422 INFO:tasks.ceph.osd.12.smithi153.stderr: 0> 2023-11-01T04:16:47.404+0000 7fb1a78cf640 -1 *** Caught signal (Aborted) ** 2023-11-01T04:16:47.422 INFO:tasks.ceph.osd.12.smithi153.stderr: in thread 7fb1a78cf640 thread_name:tp_osd_tp 2023-11-01T04:16:47.422 INFO:tasks.ceph.osd.12.smithi153.stderr: 2023-11-01T04:16:47.422 INFO:tasks.ceph.osd.12.smithi153.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:47.423 INFO:tasks.ceph.osd.12.smithi153.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb1c46ec520] 2023-11-01T04:16:47.423 INFO:tasks.ceph.osd.12.smithi153.stderr: 2: pthread_kill() 2023-11-01T04:16:47.423 INFO:tasks.ceph.osd.12.smithi153.stderr: 3: raise() 2023-11-01T04:16:47.423 INFO:tasks.ceph.osd.12.smithi153.stderr: 4: abort() 2023-11-01T04:16:47.423 INFO:tasks.ceph.osd.12.smithi153.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x560de29fda91] 2023-11-01T04:16:47.424 INFO:tasks.ceph.osd.12.smithi153.stderr: 6: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x560de2ce1c9d] 2023-11-01T04:16:47.424 INFO:tasks.ceph.osd.12.smithi153.stderr: 7: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x560de2c4c2c2] 2023-11-01T04:16:47.424 INFO:tasks.ceph.osd.12.smithi153.stderr: 8: ceph-osd(+0xd7e721) [0x560de2aa7721] 2023-11-01T04:16:47.424 INFO:tasks.ceph.osd.12.smithi153.stderr: 9: ceph-osd(+0x1695cbb) [0x560de33becbb] 2023-11-01T04:16:47.424 INFO:tasks.ceph.osd.12.smithi153.stderr: 10: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x560de2be9288] 2023-11-01T04:16:47.424 INFO:tasks.ceph.osd.12.smithi153.stderr: 11: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x560de2d1f9df] 2023-11-01T04:16:47.425 INFO:tasks.ceph.osd.12.smithi153.stderr: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x560de2b3dd23] 2023-11-01T04:16:47.425 INFO:tasks.ceph.osd.12.smithi153.stderr: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x560de2fef053] 2023-11-01T04:16:47.425 INFO:tasks.ceph.osd.12.smithi153.stderr: 14: ceph-osd(+0x12c6584) [0x560de2fef584] 2023-11-01T04:16:47.425 INFO:tasks.ceph.osd.12.smithi153.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb1c473eb43] 2023-11-01T04:16:47.425 INFO:tasks.ceph.osd.12.smithi153.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb1c47d0a00] 2023-11-01T04:16:47.425 INFO:tasks.ceph.osd.12.smithi153.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-01T04:16:47.426 INFO:tasks.ceph.osd.12.smithi153.stderr: 2023-11-01T04:16:47.426 INFO:tasks.ceph.osd.12.smithi153.stderr: -6922> 2023-11-01T04:16:44.744+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:47.426 INFO:tasks.ceph.osd.12.smithi153.stderr: -5906> 2023-11-01T04:16:45.260+0000 7fb1c0ae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 35777) UID: 0 2023-11-01T04:16:47.429 INFO:tasks.ceph.osd.12.smithi153.stderr: -53> 2023-11-01T04:16:47.392+0000 7fb1a78cf640 -1 snap_mapper.get_next_objects_to_trimNew Clone-Objects were added to Snap b after trimming was started 2023-11-01T04:16:47.429 INFO:tasks.ceph.osd.12.smithi153.stderr: -52> 2023-11-01T04:16:47.392+0000 7fb1a78cf640 -1 snap_mapper.0x7fb1a78cb750 2023-11-01T04:16:47.429 INFO:tasks.ceph.osd.12.smithi153.stderr: -2> 2023-11-01T04:16:47.396+0000 7fb1a78cf640 -1 ./src/osd/SnapMapper.cc: In function 'int SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector*)' thread 7fb1a78cf640 time 2023-11-01T04:16:47.395771+0000 2023-11-01T04:16:47.429 INFO:tasks.ceph.osd.12.smithi153.stderr:./src/osd/SnapMapper.cc: 650: ceph_abort_msg("Clone-Objects were added to a trimmed snap") 2023-11-01T04:16:47.430 INFO:tasks.ceph.osd.12.smithi153.stderr: 2023-11-01T04:16:47.430 INFO:tasks.ceph.osd.12.smithi153.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:47.430 INFO:tasks.ceph.osd.12.smithi153.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x560de29fd9dc] 2023-11-01T04:16:47.430 INFO:tasks.ceph.osd.12.smithi153.stderr: 2: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x560de2ce1c9d] 2023-11-01T04:16:47.430 INFO:tasks.ceph.osd.12.smithi153.stderr: 3: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x560de2c4c2c2] 2023-11-01T04:16:47.430 INFO:tasks.ceph.osd.12.smithi153.stderr: 4: ceph-osd(+0xd7e721) [0x560de2aa7721] 2023-11-01T04:16:47.431 INFO:tasks.ceph.osd.12.smithi153.stderr: 5: ceph-osd(+0x1695cbb) [0x560de33becbb] 2023-11-01T04:16:47.431 INFO:tasks.ceph.osd.12.smithi153.stderr: 6: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x560de2be9288] 2023-11-01T04:16:47.431 INFO:tasks.ceph.osd.12.smithi153.stderr: 7: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x560de2d1f9df] 2023-11-01T04:16:47.431 INFO:tasks.ceph.osd.12.smithi153.stderr: 8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x560de2b3dd23] 2023-11-01T04:16:47.431 INFO:tasks.ceph.osd.12.smithi153.stderr: 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x560de2fef053] 2023-11-01T04:16:47.432 INFO:tasks.ceph.osd.12.smithi153.stderr: 10: ceph-osd(+0x12c6584) [0x560de2fef584] 2023-11-01T04:16:47.432 INFO:tasks.ceph.osd.12.smithi153.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb1c473eb43] 2023-11-01T04:16:47.432 INFO:tasks.ceph.osd.12.smithi153.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb1c47d0a00] 2023-11-01T04:16:47.432 INFO:tasks.ceph.osd.12.smithi153.stderr: 2023-11-01T04:16:47.432 INFO:tasks.ceph.osd.12.smithi153.stderr: 0> 2023-11-01T04:16:47.404+0000 7fb1a78cf640 -1 *** Caught signal (Aborted) ** 2023-11-01T04:16:47.432 INFO:tasks.ceph.osd.12.smithi153.stderr: in thread 7fb1a78cf640 thread_name:tp_osd_tp 2023-11-01T04:16:47.433 INFO:tasks.ceph.osd.12.smithi153.stderr: 2023-11-01T04:16:47.433 INFO:tasks.ceph.osd.12.smithi153.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:47.433 INFO:tasks.ceph.osd.12.smithi153.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb1c46ec520] 2023-11-01T04:16:47.433 INFO:tasks.ceph.osd.12.smithi153.stderr: 2: pthread_kill() 2023-11-01T04:16:47.433 INFO:tasks.ceph.osd.12.smithi153.stderr: 3: raise() 2023-11-01T04:16:47.434 INFO:tasks.ceph.osd.12.smithi153.stderr: 4: abort() 2023-11-01T04:16:47.434 INFO:tasks.ceph.osd.12.smithi153.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x560de29fda91] 2023-11-01T04:16:47.434 INFO:tasks.ceph.osd.12.smithi153.stderr: 6: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x560de2ce1c9d] 2023-11-01T04:16:47.434 INFO:tasks.ceph.osd.12.smithi153.stderr: 7: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x560de2c4c2c2] 2023-11-01T04:16:47.434 INFO:tasks.ceph.osd.12.smithi153.stderr: 8: ceph-osd(+0xd7e721) [0x560de2aa7721] 2023-11-01T04:16:47.434 INFO:tasks.ceph.osd.12.smithi153.stderr: 9: ceph-osd(+0x1695cbb) [0x560de33becbb] 2023-11-01T04:16:47.435 INFO:tasks.ceph.osd.12.smithi153.stderr: 10: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x560de2be9288] 2023-11-01T04:16:47.435 INFO:tasks.ceph.osd.12.smithi153.stderr: 11: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x560de2d1f9df] 2023-11-01T04:16:47.435 INFO:tasks.ceph.osd.12.smithi153.stderr: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x560de2b3dd23] 2023-11-01T04:16:47.435 INFO:tasks.ceph.osd.12.smithi153.stderr: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x560de2fef053] 2023-11-01T04:16:47.435 INFO:tasks.ceph.osd.12.smithi153.stderr: 14: ceph-osd(+0x12c6584) [0x560de2fef584] 2023-11-01T04:16:47.435 INFO:tasks.ceph.osd.12.smithi153.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb1c473eb43] 2023-11-01T04:16:47.436 INFO:tasks.ceph.osd.12.smithi153.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb1c47d0a00] 2023-11-01T04:16:47.436 INFO:tasks.ceph.osd.12.smithi153.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-01T04:16:47.436 INFO:tasks.ceph.osd.12.smithi153.stderr: 2023-11-01T04:16:47.464 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:47.464+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:47.498 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:47.498 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:47.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-01T04:16:47.565 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:47.561+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:47.636 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:47.636 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:47.651 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-01T04:16:47.665 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:47.662+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:47.766 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:47.765+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:47.772 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:47.772 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:47.777 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.12 2023-11-01T04:16:47.781 INFO:tasks.rados.rados.0.smithi190.stdout:216: done (4 left) 2023-11-01T04:16:47.781 INFO:tasks.rados.rados.0.smithi190.stdout:217: rmattr oid 2 current snap is 20 2023-11-01T04:16:47.781 INFO:tasks.rados.rados.0.smithi190.stdout:217: done (4 left) 2023-11-01T04:16:47.781 INFO:tasks.rados.rados.0.smithi190.stdout:218: rmattr oid 15 current snap is 20 2023-11-01T04:16:47.781 INFO:tasks.rados.rados.0.smithi190.stdout:218: done (4 left) 2023-11-01T04:16:47.782 INFO:tasks.rados.rados.0.smithi190.stdout:219: rmattr oid 11 current snap is 20 2023-11-01T04:16:47.782 INFO:tasks.rados.rados.0.smithi190.stdout:219: done (4 left) 2023-11-01T04:16:47.782 INFO:tasks.rados.rados.0.smithi190.stdout:220: read oid 4 snap -1 2023-11-01T04:16:47.782 INFO:tasks.rados.rados.0.smithi190.stdout:220: expect (ObjNum 3 snap 0 seq_num 3) 2023-11-01T04:16:47.782 INFO:tasks.rados.rados.0.smithi190.stdout:221: read oid 18 snap -1 2023-11-01T04:16:47.783 INFO:tasks.rados.rados.0.smithi190.stdout:221: expect deleted 2023-11-01T04:16:47.783 INFO:tasks.rados.rados.0.smithi190.stdout:222: setattr oid 25 current snap is 20 2023-11-01T04:16:47.783 INFO:tasks.rados.rados.0.smithi190.stdout:211: finishing write tid 4 to smithi19039307-9 2023-11-01T04:16:47.783 INFO:tasks.rados.rados.0.smithi190.stdout:210: finishing write tid 5 to smithi19039307-37 2023-11-01T04:16:47.783 INFO:tasks.rados.rados.0.smithi190.stdout:210: oid 37 updating version 0 to 44 2023-11-01T04:16:47.784 INFO:tasks.rados.rados.0.smithi190.stdout:210: oid 37 version 44 is already newer than 43 2023-11-01T04:16:47.784 INFO:tasks.rados.rados.0.smithi190.stdout:210: oid 37 updating version 44 to 45 2023-11-01T04:16:47.784 INFO:tasks.rados.rados.0.smithi190.stdout:210: oid 37 updating version 45 to 46 2023-11-01T04:16:47.784 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 37 v 46 (ObjNum 87 snap 20 seq_num 87) dirty exists 2023-11-01T04:16:47.784 INFO:tasks.rados.rados.0.smithi190.stdout:210: left oid 37 (ObjNum 87 snap 20 seq_num 87) 2023-11-01T04:16:47.785 INFO:tasks.rados.rados.0.smithi190.stdout:211: finishing write tid 5 to smithi19039307-9 2023-11-01T04:16:47.785 INFO:tasks.rados.rados.0.smithi190.stdout:211: oid 9 updating version 0 to 17 2023-11-01T04:16:47.785 INFO:tasks.rados.rados.0.smithi190.stdout:211: oid 9 version 17 is already newer than 14 2023-11-01T04:16:47.785 INFO:tasks.rados.rados.0.smithi190.stdout:211: oid 9 version 17 is already newer than 16 2023-11-01T04:16:47.785 INFO:tasks.rados.rados.0.smithi190.stdout:211: oid 9 version 17 is already newer than 15 2023-11-01T04:16:47.785 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 9 v 17 (ObjNum 88 snap 20 seq_num 88) dirty exists 2023-11-01T04:16:47.786 INFO:tasks.rados.rados.0.smithi190.stdout:211: left oid 9 (ObjNum 88 snap 20 seq_num 88) 2023-11-01T04:16:47.786 INFO:tasks.rados.rados.0.smithi190.stdout:214: finishing write tid 1 to smithi19039307-19 2023-11-01T04:16:47.786 INFO:tasks.rados.rados.0.smithi190.stdout:214: finishing write tid 2 to smithi19039307-19 2023-11-01T04:16:47.786 INFO:tasks.rados.rados.0.smithi190.stdout:214: finishing write tid 3 to smithi19039307-19 2023-11-01T04:16:47.786 INFO:tasks.rados.rados.0.smithi190.stdout:214: finishing write tid 4 to smithi19039307-19 2023-11-01T04:16:47.787 INFO:tasks.rados.rados.0.smithi190.stdout:210: done (6 left) 2023-11-01T04:16:47.787 INFO:tasks.rados.rados.0.smithi190.stdout:211: done (5 left) 2023-11-01T04:16:47.787 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 30 current snap is 20 2023-11-01T04:16:47.788 INFO:tasks.rados.rados.0.smithi190.stdout:223: seq_num 90 ranges {491520=753664,1245184=720896,1966080=49152} 2023-11-01T04:16:47.793 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-11-01T04:16:47.794 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-01T04:16:47.806 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:47.807 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-01T04:16:47.807 INFO:tasks.rados.rados.0.smithi190.stdout:223: writing smithi19039307-30 from 491520 to 1245184 tid 1 2023-11-01T04:16:47.818 INFO:tasks.rados.rados.0.smithi190.stdout:223: writing smithi19039307-30 from 1245184 to 1966080 tid 2 2023-11-01T04:16:47.818 INFO:tasks.rados.rados.0.smithi190.stdout:223: writing smithi19039307-30 from 1966080 to 2015232 tid 3 2023-11-01T04:16:47.819 INFO:tasks.rados.rados.0.smithi190.stdout:224: copy_from oid 2 from oid 34 current snap is 20 2023-11-01T04:16:47.819 INFO:tasks.rados.rados.0.smithi190.stdout:215: finishing copy_from racing read to smithi19039307-36 2023-11-01T04:16:47.819 INFO:tasks.rados.rados.0.smithi190.stdout:225: snap_create 2023-11-01T04:16:47.919 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:47.919 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-01T04:16:47.933 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:47.933 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2023-11-01T04:16:47.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:47.933 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.4s5 68/31 ECSubWrite(tid=80, reqid=client.4559.0:550, at_version=68'34, trim_to=0'0, roll_forward_to=67'29))", 2023-11-01T04:16:47.934 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:47.925362+0000", 2023-11-01T04:16:47.934 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0093655379999999996, 2023-11-01T04:16:47.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094030629999999997, 2023-11-01T04:16:47.934 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:47.934 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:47.935 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:47.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:47.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:47.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.925362+0000", 2023-11-01T04:16:47.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:47.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:47.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:47.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:47.936 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.925362+0000", 2023-11-01T04:16:47.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:47.936 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:47.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:47.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:47.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.925363+0000", 2023-11-01T04:16:47.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.474e-06 2023-11-01T04:16:47.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:47.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:47.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:47.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.925392+0000", 2023-11-01T04:16:47.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9011999999999999e-05 2023-11-01T04:16:47.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:47.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:47.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:47.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.925420+0000", 2023-11-01T04:16:47.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7322e-05 2023-11-01T04:16:47.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:47.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:47.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:47.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.925437+0000", 2023-11-01T04:16:47.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7839999999999999e-05 2023-11-01T04:16:47.940 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:47.940 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:47.940 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:47.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:47.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:47.941 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 68/31 ECSubWriteReply(tid=36, last_complete=68'21, committed=1, applied=1))", 2023-11-01T04:16:47.941 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:47.893907+0000", 2023-11-01T04:16:47.941 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.040820452, 2023-11-01T04:16:47.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.040942325000000002, 2023-11-01T04:16:47.942 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:47.942 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:47.942 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:47.942 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:47.942 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:47.942 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.893907+0000", 2023-11-01T04:16:47.943 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:47.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:47.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:47.943 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:47.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.893907+0000", 2023-11-01T04:16:47.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:47.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:47.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:47.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:47.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.893908+0000", 2023-11-01T04:16:47.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3399999999999998e-07 2023-11-01T04:16:47.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:47.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:47.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:47.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.893913+0000", 2023-11-01T04:16:47.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.305e-06 2023-11-01T04:16:47.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:47.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:47.946 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:47.946 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.893930+0000", 2023-11-01T04:16:47.946 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6932e-05 2023-11-01T04:16:47.946 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:47.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:47.947 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:47.947 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.893938+0000", 2023-11-01T04:16:47.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4279999999999996e-06 2023-11-01T04:16:47.947 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:47.947 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:47.947 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:47.948 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:47.948 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2023-11-01T04:16:47.948 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 2 2023-11-01T04:16:47.948 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:47.949 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-01T04:16:47.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-11-01T04:16:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-01T04:16:47.951 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-01T04:16:47.968 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:47.965+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:48.008 INFO:tasks.rados.rados.0.smithi190.stdout:215: finishing copy_from to smithi19039307-36 2023-11-01T04:16:48.008 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 36 v 8 (ObjNum 44 snap 0 seq_num 44) dirty exists 2023-11-01T04:16:48.069 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:48.069+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:48.095 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 25 v 44 (ObjNum 76 snap 13 seq_num 76) dirty exists 2023-11-01T04:16:48.096 INFO:tasks.rados.rados.0.smithi190.stdout:220: expect (ObjNum 3 snap 0 seq_num 3) 2023-11-01T04:16:48.111 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2023-11-01T04:16:48.111 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2023-11-01T04:16:48.111 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:48.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-11-01T04:16:48.170 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:48.167+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:48.267 INFO:tasks.ceph.osd.12.smithi153.stderr:daemon-helper: command crashed with signal 6 2023-11-01T04:16:48.271 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:48.272+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:48.289 INFO:tasks.rados.rados.0.smithi190.stdout:223: finishing write tid 1 to smithi19039307-30 2023-11-01T04:16:48.299 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:48.300 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2023-11-01T04:16:48.300 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2023-11-01T04:16:48.300 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2023-11-01T04:16:48.300 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.300 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:136 3.0s0 3:0e42a959:::smithi19039307-46:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:48.301 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:15.181743+0000", 2023-11-01T04:16:48.301 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 33.116150861000001, 2023-11-01T04:16:48.301 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40000987799999999, 2023-11-01T04:16:48.301 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.301 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:48.301 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:48.302 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:48.302 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:48.302 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 136 2023-11-01T04:16:48.302 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.302 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.303 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.303 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.303 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.181743+0000", 2023-11-01T04:16:48.303 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.304 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.304 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.181743+0000", 2023-11-01T04:16:48.304 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.304 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.305 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.305 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.181745+0000", 2023-11-01T04:16:48.305 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5799999999999999e-06 2023-11-01T04:16:48.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.305 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.306 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.306 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.182798+0000", 2023-11-01T04:16:48.306 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001052648 2023-11-01T04:16:48.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.306 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.307 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.182830+0000", 2023-11-01T04:16:48.307 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1786999999999999e-05 2023-11-01T04:16:48.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.307 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.307 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.308 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.282919+0000", 2023-11-01T04:16:48.308 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10008908599999999 2023-11-01T04:16:48.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.308 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.308 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283049+0000", 2023-11-01T04:16:48.309 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013068500000000001 2023-11-01T04:16:48.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.309 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.309 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:48.309 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283434+0000", 2023-11-01T04:16:48.310 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038438399999999998 2023-11-01T04:16:48.310 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.310 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.310 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:48.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.286060+0000", 2023-11-01T04:16:48.311 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026267180000000001 2023-11-01T04:16:48.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.311 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.311 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:48.311 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.290497+0000", 2023-11-01T04:16:48.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044367979999999996 2023-11-01T04:16:48.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.312 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.312 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:48.312 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.581619+0000", 2023-11-01T04:16:48.312 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29112156700000003 2023-11-01T04:16:48.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.313 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.313 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.313 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.581752+0000", 2023-11-01T04:16:48.313 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013362500000000001 2023-11-01T04:16:48.314 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.314 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.314 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.314 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.315 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:137 3.0s0 3:0e42a959:::smithi19039307-46:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:48.315 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:15.282962+0000", 2023-11-01T04:16:48.315 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 33.014931339999997, 2023-11-01T04:16:48.315 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.32685867899999999, 2023-11-01T04:16:48.315 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.315 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 137 2023-11-01T04:16:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.282962+0000", 2023-11-01T04:16:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.318 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.318 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.282962+0000", 2023-11-01T04:16:48.318 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.318 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.319 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.319 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.282964+0000", 2023-11-01T04:16:48.319 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9269999999999999e-06 2023-11-01T04:16:48.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.319 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.320 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.320 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.282975+0000", 2023-11-01T04:16:48.320 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0883000000000001e-05 2023-11-01T04:16:48.320 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.320 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.320 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.321 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.282997+0000", 2023-11-01T04:16:48.321 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1897999999999999e-05 2023-11-01T04:16:48.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.321 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.321 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.321 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283012+0000", 2023-11-01T04:16:48.322 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5682999999999999e-05 2023-11-01T04:16:48.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.322 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.322 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.322 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.287942+0000", 2023-11-01T04:16:48.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049297919999999997 2023-11-01T04:16:48.323 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.323 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.323 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:48.323 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.288248+0000", 2023-11-01T04:16:48.323 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030577300000000001 2023-11-01T04:16:48.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.324 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.324 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:48.324 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.289532+0000", 2023-11-01T04:16:48.324 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012844060000000001 2023-11-01T04:16:48.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.325 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.325 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:48.325 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.290865+0000", 2023-11-01T04:16:48.325 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013323320000000001 2023-11-01T04:16:48.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.326 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:48.326 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.609653+0000", 2023-11-01T04:16:48.326 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31878868999999999 2023-11-01T04:16:48.326 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.326 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.327 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.327 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.609821+0000", 2023-11-01T04:16:48.327 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000167295 2023-11-01T04:16:48.327 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.327 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.327 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.328 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:138 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:48.328 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:15.283030+0000", 2023-11-01T04:16:48.328 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 33.014863349999999, 2023-11-01T04:16:48.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52972434099999999, 2023-11-01T04:16:48.329 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.329 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:48.329 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:48.329 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:48.329 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:48.330 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 138 2023-11-01T04:16:48.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.330 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.330 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283030+0000", 2023-11-01T04:16:48.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283030+0000", 2023-11-01T04:16:48.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283031+0000", 2023-11-01T04:16:48.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9600000000000008e-07 2023-11-01T04:16:48.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.333 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.333 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283035+0000", 2023-11-01T04:16:48.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7299999999999999e-06 2023-11-01T04:16:48.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283046+0000", 2023-11-01T04:16:48.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1249e-05 2023-11-01T04:16:48.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283055+0000", 2023-11-01T04:16:48.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2660000000000007e-06 2023-11-01T04:16:48.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.336 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.290158+0000", 2023-11-01T04:16:48.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0071024260000000002 2023-11-01T04:16:48.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.336 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for rw locks", 2023-11-01T04:16:48.337 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.290320+0000", 2023-11-01T04:16:48.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000161886 2023-11-01T04:16:48.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.337 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.337 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.337 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.610629+0000", 2023-11-01T04:16:48.338 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.32030918600000002 2023-11-01T04:16:48.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.338 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.338 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:48.338 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.610871+0000", 2023-11-01T04:16:48.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000242124 2023-11-01T04:16:48.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.339 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.812754+0000", 2023-11-01T04:16:48.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20188347800000001 2023-11-01T04:16:48.340 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.340 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.340 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.340 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:161 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~770048,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:48.341 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.318323+0000", 2023-11-01T04:16:48.341 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.97957061, 2023-11-01T04:16:48.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20523274599999999, 2023-11-01T04:16:48.341 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.341 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:48.341 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:48.342 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:48.342 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:48.342 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 161 2023-11-01T04:16:48.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.342 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.343 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.343 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318323+0000", 2023-11-01T04:16:48.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318323+0000", 2023-11-01T04:16:48.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318327+0000", 2023-11-01T04:16:48.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.726e-06 2023-11-01T04:16:48.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318354+0000", 2023-11-01T04:16:48.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7404999999999999e-05 2023-11-01T04:16:48.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318523+0000", 2023-11-01T04:16:48.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016939900000000001 2023-11-01T04:16:48.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318550+0000", 2023-11-01T04:16:48.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6338000000000001e-05 2023-11-01T04:16:48.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318678+0000", 2023-11-01T04:16:48.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000127899 2023-11-01T04:16:48.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:48.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318814+0000", 2023-11-01T04:16:48.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013687399999999999 2023-11-01T04:16:48.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.350 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.350 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.523556+0000", 2023-11-01T04:16:48.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20474110500000001 2023-11-01T04:16:48.350 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.351 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.351 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.351 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:162 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~770048,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:48.351 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.318603+0000", 2023-11-01T04:16:48.352 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.979290557999999, 2023-11-01T04:16:48.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.206224885, 2023-11-01T04:16:48.352 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.352 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:48.352 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:48.353 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:48.353 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:48.353 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 162 2023-11-01T04:16:48.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.353 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318603+0000", 2023-11-01T04:16:48.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318603+0000", 2023-11-01T04:16:48.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318605+0000", 2023-11-01T04:16:48.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.903e-06 2023-11-01T04:16:48.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318614+0000", 2023-11-01T04:16:48.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6360000000000006e-06 2023-11-01T04:16:48.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318637+0000", 2023-11-01T04:16:48.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2617999999999999e-05 2023-11-01T04:16:48.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318657+0000", 2023-11-01T04:16:48.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9551999999999998e-05 2023-11-01T04:16:48.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.319458+0000", 2023-11-01T04:16:48.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00080107500000000005 2023-11-01T04:16:48.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:48.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.319692+0000", 2023-11-01T04:16:48.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000234675 2023-11-01T04:16:48.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.524828+0000", 2023-11-01T04:16:48.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20513542600000001 2023-11-01T04:16:48.361 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.362 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.362 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.362 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:163 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~770048,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:48.362 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.318690+0000", 2023-11-01T04:16:48.363 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.979203764000001, 2023-11-01T04:16:48.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.309365155, 2023-11-01T04:16:48.363 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.363 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:48.363 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:48.364 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:48.364 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:48.364 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 163 2023-11-01T04:16:48.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.364 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318690+0000", 2023-11-01T04:16:48.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318690+0000", 2023-11-01T04:16:48.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318691+0000", 2023-11-01T04:16:48.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.263e-06 2023-11-01T04:16:48.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318698+0000", 2023-11-01T04:16:48.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7900000000000002e-06 2023-11-01T04:16:48.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.318716+0000", 2023-11-01T04:16:48.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8128e-05 2023-11-01T04:16:48.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.418889+0000", 2023-11-01T04:16:48.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100173525 2023-11-01T04:16:48.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.524945+0000", 2023-11-01T04:16:48.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.106055714 2023-11-01T04:16:48.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:48.371 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.525089+0000", 2023-11-01T04:16:48.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014418800000000001 2023-11-01T04:16:48.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.371 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.628055+0000", 2023-11-01T04:16:48.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.102965547 2023-11-01T04:16:48.372 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.372 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.372 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.373 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 36/31 ECSubReadReply(tid=8, attrs_read=0))", 2023-11-01T04:16:48.373 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.321399+0000", 2023-11-01T04:16:48.373 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.976494706, 2023-11-01T04:16:48.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20100272, 2023-11-01T04:16:48.374 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.374 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:48.374 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.321399+0000", 2023-11-01T04:16:48.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.321399+0000", 2023-11-01T04:16:48.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.321400+0000", 2023-11-01T04:16:48.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7430000000000001e-06 2023-11-01T04:16:48.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322320+0000", 2023-11-01T04:16:48.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000919454 2023-11-01T04:16:48.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322347+0000", 2023-11-01T04:16:48.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6758e-05 2023-11-01T04:16:48.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322364+0000", 2023-11-01T04:16:48.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6901999999999998e-05 2023-11-01T04:16:48.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.522338+0000", 2023-11-01T04:16:48.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.199974127 2023-11-01T04:16:48.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.522401+0000", 2023-11-01T04:16:48.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3736000000000001e-05 2023-11-01T04:16:48.381 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.381 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.381 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.382 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 36/31 ECSubReadReply(tid=9, attrs_read=0))", 2023-11-01T04:16:48.382 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.322400+0000", 2023-11-01T04:16:48.382 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.975493791000002, 2023-11-01T04:16:48.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20014336599999999, 2023-11-01T04:16:48.382 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.383 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:48.383 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322400+0000", 2023-11-01T04:16:48.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322400+0000", 2023-11-01T04:16:48.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322401+0000", 2023-11-01T04:16:48.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1599999999999999e-06 2023-11-01T04:16:48.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322751+0000", 2023-11-01T04:16:48.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000349897 2023-11-01T04:16:48.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322777+0000", 2023-11-01T04:16:48.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6171999999999999e-05 2023-11-01T04:16:48.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.322791+0000", 2023-11-01T04:16:48.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3939e-05 2023-11-01T04:16:48.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.522487+0000", 2023-11-01T04:16:48.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.199695822 2023-11-01T04:16:48.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.522543+0000", 2023-11-01T04:16:48.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6375999999999997e-05 2023-11-01T04:16:48.390 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.390 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.390 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.391 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.8s3 41/31 ECSubWrite(tid=22, reqid=osd.5.0:21, at_version=0'0, trim_to=0'0, roll_forward_to=41'12))", 2023-11-01T04:16:48.391 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.441913+0000", 2023-11-01T04:16:48.391 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 26.855980398, 2023-11-01T04:16:48.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20159738899999999, 2023-11-01T04:16:48.391 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.391 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:48.392 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441913+0000", 2023-11-01T04:16:48.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441913+0000", 2023-11-01T04:16:48.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441914+0000", 2023-11-01T04:16:48.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8999999999999995e-07 2023-11-01T04:16:48.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441929+0000", 2023-11-01T04:16:48.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5358000000000001e-05 2023-11-01T04:16:48.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.441947+0000", 2023-11-01T04:16:48.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7323e-05 2023-11-01T04:16:48.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.542038+0000", 2023-11-01T04:16:48.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10009142 2023-11-01T04:16:48.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.642225+0000", 2023-11-01T04:16:48.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100186991 2023-11-01T04:16:48.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:48.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.642241+0000", 2023-11-01T04:16:48.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6302000000000001e-05 2023-11-01T04:16:48.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:48.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.643431+0000", 2023-11-01T04:16:48.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011899359999999999 2023-11-01T04:16:48.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.643510+0000", 2023-11-01T04:16:48.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9169e-05 2023-11-01T04:16:48.400 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.401 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.401 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.401 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.es5 41/31 ECSubWrite(tid=37, reqid=osd.11.0:36, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2023-11-01T04:16:48.401 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.645104+0000", 2023-11-01T04:16:48.402 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 26.652789330000001, 2023-11-01T04:16:48.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.200759945, 2023-11-01T04:16:48.402 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.402 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:48.402 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.645104+0000", 2023-11-01T04:16:48.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.645104+0000", 2023-11-01T04:16:48.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.645106+0000", 2023-11-01T04:16:48.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.623e-06 2023-11-01T04:16:48.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.645133+0000", 2023-11-01T04:16:48.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6882000000000001e-05 2023-11-01T04:16:48.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.645165+0000", 2023-11-01T04:16:48.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2787999999999998e-05 2023-11-01T04:16:48.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.745255+0000", 2023-11-01T04:16:48.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100089689 2023-11-01T04:16:48.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.745329+0000", 2023-11-01T04:16:48.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4295000000000006e-05 2023-11-01T04:16:48.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:48.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.745338+0000", 2023-11-01T04:16:48.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9830000000000002e-06 2023-11-01T04:16:48.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:48.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.845832+0000", 2023-11-01T04:16:48.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100493597 2023-11-01T04:16:48.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.845864+0000", 2023-11-01T04:16:48.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2088000000000002e-05 2023-11-01T04:16:48.411 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.411 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.412 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.412 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s1 46/31 ECSubWrite(tid=236, reqid=client.4559.0:330, at_version=46'36, trim_to=0'0, roll_forward_to=45'33))", 2023-11-01T04:16:48.412 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:26.290271+0000", 2023-11-01T04:16:48.412 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 22.007622606999998, 2023-11-01T04:16:48.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30111696100000002, 2023-11-01T04:16:48.413 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.413 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:48.413 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.290271+0000", 2023-11-01T04:16:48.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.290271+0000", 2023-11-01T04:16:48.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.290272+0000", 2023-11-01T04:16:48.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.04e-06 2023-11-01T04:16:48.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.290503+0000", 2023-11-01T04:16:48.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000231635 2023-11-01T04:16:48.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.290527+0000", 2023-11-01T04:16:48.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3589000000000001e-05 2023-11-01T04:16:48.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.390628+0000", 2023-11-01T04:16:48.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10010121600000001 2023-11-01T04:16:48.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.390905+0000", 2023-11-01T04:16:48.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000276586 2023-11-01T04:16:48.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:48.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.390935+0000", 2023-11-01T04:16:48.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9876e-05 2023-11-01T04:16:48.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:48.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.591318+0000", 2023-11-01T04:16:48.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.200382906 2023-11-01T04:16:48.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.591388+0000", 2023-11-01T04:16:48.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0112999999999998e-05 2023-11-01T04:16:48.422 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.422 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.422 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.423 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.2s1 46/31 ECSubWrite(tid=237, reqid=client.4559.0:331, at_version=46'37, trim_to=0'0, roll_forward_to=45'33))", 2023-11-01T04:16:48.423 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:26.390678+0000", 2023-11-01T04:16:48.423 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.907215739000002, 2023-11-01T04:16:48.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20077771999999999, 2023-11-01T04:16:48.423 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.424 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:48.424 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.390678+0000", 2023-11-01T04:16:48.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.390678+0000", 2023-11-01T04:16:48.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.390679+0000", 2023-11-01T04:16:48.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4470000000000001e-06 2023-11-01T04:16:48.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.390718+0000", 2023-11-01T04:16:48.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9121e-05 2023-11-01T04:16:48.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.390739+0000", 2023-11-01T04:16:48.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0496000000000002e-05 2023-11-01T04:16:48.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.390752+0000", 2023-11-01T04:16:48.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3465000000000001e-05 2023-11-01T04:16:48.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.392434+0000", 2023-11-01T04:16:48.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016820170000000001 2023-11-01T04:16:48.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:48.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.392463+0000", 2023-11-01T04:16:48.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9261999999999999e-05 2023-11-01T04:16:48.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:48.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.591396+0000", 2023-11-01T04:16:48.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.198932991 2023-11-01T04:16:48.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.591455+0000", 2023-11-01T04:16:48.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8921000000000001e-05 2023-11-01T04:16:48.432 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.433 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.433 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.433 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpRead(3.2s1 46/31 ECSubRead(tid=240, to_read={3:4a13469b:::smithi19039307-24:head=0,561152,0}, subchunks={3:4a13469b:::smithi19039307-24:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:48.433 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:26.390997+0000", 2023-11-01T04:16:48.434 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.906896695, 2023-11-01T04:16:48.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20031143700000001, 2023-11-01T04:16:48.434 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.434 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:48.434 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.390997+0000", 2023-11-01T04:16:48.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.390997+0000", 2023-11-01T04:16:48.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.390997+0000", 2023-11-01T04:16:48.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8299999999999996e-07 2023-11-01T04:16:48.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.391007+0000", 2023-11-01T04:16:48.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.55e-06 2023-11-01T04:16:48.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.391029+0000", 2023-11-01T04:16:48.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2408e-05 2023-11-01T04:16:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.391056+0000", 2023-11-01T04:16:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6109e-05 2023-11-01T04:16:48.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.591042+0000", 2023-11-01T04:16:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.19998616999999999 2023-11-01T04:16:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:26.591308+0000", 2023-11-01T04:16:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026641700000000002 2023-11-01T04:16:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.441 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.442 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.442 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:444 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~770048,checksum in=4b,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2023-11-01T04:16:48.442 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.143530+0000", 2023-11-01T04:16:48.442 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.1543630650000001, 2023-11-01T04:16:48.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40681932599999998, 2023-11-01T04:16:48.443 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.443 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:48.443 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:48.443 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:48.443 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:48.444 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 444 2023-11-01T04:16:48.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.444 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.143530+0000", 2023-11-01T04:16:48.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.143530+0000", 2023-11-01T04:16:48.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.143534+0000", 2023-11-01T04:16:48.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2940000000000001e-06 2023-11-01T04:16:48.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.143566+0000", 2023-11-01T04:16:48.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2098999999999999e-05 2023-11-01T04:16:48.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.143602+0000", 2023-11-01T04:16:48.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6301999999999999e-05 2023-11-01T04:16:48.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243774+0000", 2023-11-01T04:16:48.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100171959 2023-11-01T04:16:48.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344071+0000", 2023-11-01T04:16:48.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100297211 2023-11-01T04:16:48.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2023-11-01T04:16:48.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344083+0000", 2023-11-01T04:16:48.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.203e-05 2023-11-01T04:16:48.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344705+0000", 2023-11-01T04:16:48.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000621515 2023-11-01T04:16:48.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:48.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344857+0000", 2023-11-01T04:16:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000152008 2023-11-01T04:16:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.550350+0000", 2023-11-01T04:16:48.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.205492908 2023-11-01T04:16:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:445 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~770048,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2023-11-01T04:16:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.243815+0000", 2023-11-01T04:16:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.0540787129999991, 2023-11-01T04:16:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40799661599999998, 2023-11-01T04:16:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.455 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:48.456 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:48.456 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:48.456 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:48.456 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 445 2023-11-01T04:16:48.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.456 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243815+0000", 2023-11-01T04:16:48.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243815+0000", 2023-11-01T04:16:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243816+0000", 2023-11-01T04:16:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5999999999999999e-06 2023-11-01T04:16:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243826+0000", 2023-11-01T04:16:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4029999999999996e-06 2023-11-01T04:16:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243844+0000", 2023-11-01T04:16:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8076000000000001e-05 2023-11-01T04:16:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243855+0000", 2023-11-01T04:16:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.17e-05 2023-11-01T04:16:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344243+0000", 2023-11-01T04:16:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100387278 2023-11-01T04:16:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344254+0000", 2023-11-01T04:16:48.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1e-05 2023-11-01T04:16:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.347410+0000", 2023-11-01T04:16:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031561829999999999 2023-11-01T04:16:48.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:48.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.347553+0000", 2023-11-01T04:16:48.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000142779 2023-11-01T04:16:48.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.651811+0000", 2023-11-01T04:16:48.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30425859700000002 2023-11-01T04:16:48.466 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.466 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.467 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.467 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:446 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~770048,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2023-11-01T04:16:48.467 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.243920+0000", 2023-11-01T04:16:48.467 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 9.0539735300000004, 2023-11-01T04:16:48.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.408629468, 2023-11-01T04:16:48.468 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.468 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:48.468 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:48.468 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:48.468 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:48.469 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 446 2023-11-01T04:16:48.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.469 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243920+0000", 2023-11-01T04:16:48.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243920+0000", 2023-11-01T04:16:48.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243921+0000", 2023-11-01T04:16:48.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3399999999999998e-07 2023-11-01T04:16:48.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243926+0000", 2023-11-01T04:16:48.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.096e-06 2023-11-01T04:16:48.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243941+0000", 2023-11-01T04:16:48.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5569e-05 2023-11-01T04:16:48.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.474 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243951+0000", 2023-11-01T04:16:48.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4499999999999993e-06 2023-11-01T04:16:48.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448384+0000", 2023-11-01T04:16:48.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20443328799999999 2023-11-01T04:16:48.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:48.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448507+0000", 2023-11-01T04:16:48.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012274000000000001 2023-11-01T04:16:48.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.652549+0000", 2023-11-01T04:16:48.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20404249099999999 2023-11-01T04:16:48.477 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.477 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.477 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.478 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 59/31 ECSubReadReply(tid=27, attrs_read=0))", 2023-11-01T04:16:48.478 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.449428+0000", 2023-11-01T04:16:48.478 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 8.8484657319999993, 2023-11-01T04:16:48.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.201701932, 2023-11-01T04:16:48.478 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.479 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:48.479 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449428+0000", 2023-11-01T04:16:48.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449428+0000", 2023-11-01T04:16:48.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449428+0000", 2023-11-01T04:16:48.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.16e-07 2023-11-01T04:16:48.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449666+0000", 2023-11-01T04:16:48.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023703099999999999 2023-11-01T04:16:48.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449678+0000", 2023-11-01T04:16:48.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2687999999999999e-05 2023-11-01T04:16:48.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.449688+0000", 2023-11-01T04:16:48.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.482e-06 2023-11-01T04:16:48.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.651057+0000", 2023-11-01T04:16:48.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20136971200000001 2023-11-01T04:16:48.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.651130+0000", 2023-11-01T04:16:48.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2203e-05 2023-11-01T04:16:48.486 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.486 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.486 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.487 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 59/31 ECSubReadReply(tid=26, attrs_read=0))", 2023-11-01T04:16:48.487 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.451250+0000", 2023-11-01T04:16:48.487 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 8.8466437810000009, 2023-11-01T04:16:48.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20058273900000001, 2023-11-01T04:16:48.487 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.487 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:48.488 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.488 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.488 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451250+0000", 2023-11-01T04:16:48.488 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451250+0000", 2023-11-01T04:16:48.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451251+0000", 2023-11-01T04:16:48.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.057e-06 2023-11-01T04:16:48.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451459+0000", 2023-11-01T04:16:48.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020849700000000001 2023-11-01T04:16:48.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451476+0000", 2023-11-01T04:16:48.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6490999999999998e-05 2023-11-01T04:16:48.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451487+0000", 2023-11-01T04:16:48.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1094000000000001e-05 2023-11-01T04:16:48.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.651178+0000", 2023-11-01T04:16:48.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.199691322 2023-11-01T04:16:48.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.651832+0000", 2023-11-01T04:16:48.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000654278 2023-11-01T04:16:48.495 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.495 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.495 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.495 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 59/31 ECSubReadReply(tid=27, attrs_read=0))", 2023-11-01T04:16:48.496 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.451503+0000", 2023-11-01T04:16:48.496 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 8.8463907109999997, 2023-11-01T04:16:48.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20045005699999999, 2023-11-01T04:16:48.496 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.496 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:48.496 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451503+0000", 2023-11-01T04:16:48.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451503+0000", 2023-11-01T04:16:48.498 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.499 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.499 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451503+0000", 2023-11-01T04:16:48.499 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0400000000000002e-07 2023-11-01T04:16:48.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.499 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.499 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451679+0000", 2023-11-01T04:16:48.500 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017560000000000001 2023-11-01T04:16:48.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.500 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451711+0000", 2023-11-01T04:16:48.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2140000000000001e-05 2023-11-01T04:16:48.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.451721+0000", 2023-11-01T04:16:48.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0052e-05 2023-11-01T04:16:48.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.651895+0000", 2023-11-01T04:16:48.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.200173876 2023-11-01T04:16:48.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.651953+0000", 2023-11-01T04:16:48.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7985000000000001e-05 2023-11-01T04:16:48.504 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.504 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.504 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.504 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpRead(3.2s1 62/31 ECSubRead(tid=300, to_read={3:4174a5c9:::smithi19039307-16:head=0,4096,0}, subchunks={3:4174a5c9:::smithi19039307-16:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:48.505 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:42.672536+0000", 2023-11-01T04:16:48.505 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.6253574769999997, 2023-11-01T04:16:48.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20061863999999999, 2023-11-01T04:16:48.505 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.505 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:48.505 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.506 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.672536+0000", 2023-11-01T04:16:48.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.672536+0000", 2023-11-01T04:16:48.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.672538+0000", 2023-11-01T04:16:48.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7600000000000001e-06 2023-11-01T04:16:48.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.672554+0000", 2023-11-01T04:16:48.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6793e-05 2023-11-01T04:16:48.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.672588+0000", 2023-11-01T04:16:48.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3331999999999999e-05 2023-11-01T04:16:48.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.772737+0000", 2023-11-01T04:16:48.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10014888299999999 2023-11-01T04:16:48.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.512 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.872988+0000", 2023-11-01T04:16:48.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10025161 2023-11-01T04:16:48.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.512 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.873155+0000", 2023-11-01T04:16:48.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000166262 2023-11-01T04:16:48.513 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.513 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.513 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.513 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.514 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.514 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:48.515 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-01T04:16:48.516 INFO:tasks.rados.rados.0.smithi190.stdout:223: finishing write tid 2 to smithi19039307-30 2023-11-01T04:16:48.517 INFO:tasks.rados.rados.0.smithi190.stdout:223: finishing write tid 3 to smithi19039307-30 2023-11-01T04:16:48.517 INFO:tasks.rados.rados.0.smithi190.stdout:223: finishing write tid 4 to smithi19039307-30 2023-11-01T04:16:48.517 INFO:tasks.rados.rados.0.smithi190.stdout:224: finishing copy_from racing read to smithi19039307-2 2023-11-01T04:16:48.518 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:48.515+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:48.616 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:48.613+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:48.631 INFO:tasks.rados.rados.0.smithi190.stdout:223: finishing write tid 5 to smithi19039307-30 2023-11-01T04:16:48.631 INFO:tasks.rados.rados.0.smithi190.stdout:223: oid 30 updating version 0 to 32 2023-11-01T04:16:48.632 INFO:tasks.rados.rados.0.smithi190.stdout:223: oid 30 updating version 32 to 34 2023-11-01T04:16:48.632 INFO:tasks.rados.rados.0.smithi190.stdout:223: oid 30 version 34 is already newer than 33 2023-11-01T04:16:48.632 INFO:tasks.rados.rados.0.smithi190.stdout:223: oid 30 version 34 is already newer than 31 2023-11-01T04:16:48.632 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 30 v 34 (ObjNum 90 snap 20 seq_num 90) dirty exists 2023-11-01T04:16:48.632 INFO:tasks.rados.rados.0.smithi190.stdout:223: left oid 30 (ObjNum 90 snap 20 seq_num 90) 2023-11-01T04:16:48.650 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:48.650 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:48.650 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2023-11-01T04:16:48.650 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:48.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-01T04:16:48.711 INFO:tasks.rados.rados.0.smithi190.stdout:224: finishing copy_from to smithi19039307-2 2023-11-01T04:16:48.711 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 2 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2023-11-01T04:16:48.717 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:48.713+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:noscrub is unset 2023-11-01T04:16:48.787 INFO:tasks.rados.rados.0.smithi190.stdout:215: done (7 left) 2023-11-01T04:16:48.787 INFO:tasks.rados.rados.0.smithi190.stdout:220: done (6 left) 2023-11-01T04:16:48.787 INFO:tasks.rados.rados.0.smithi190.stdout:221: done (5 left) 2023-11-01T04:16:48.787 INFO:tasks.rados.rados.0.smithi190.stdout:222: done (4 left) 2023-11-01T04:16:48.787 INFO:tasks.rados.rados.0.smithi190.stdout:223: done (3 left) 2023-11-01T04:16:48.788 INFO:tasks.rados.rados.0.smithi190.stdout:224: done (2 left) 2023-11-01T04:16:48.788 INFO:tasks.rados.rados.0.smithi190.stdout:225: done (1 left) 2023-11-01T04:16:48.788 INFO:tasks.rados.rados.0.smithi190.stdout:226: delete oid 40 current snap is 21 2023-11-01T04:16:48.788 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:48.789 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:48.789 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2023-11-01T04:16:48.789 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2023-11-01T04:16:48.789 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:48.803 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-01T04:16:48.818 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:48.817+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:48.919 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:48.915+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:48.929 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:48.929 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2023-11-01T04:16:48.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2023-11-01T04:16:48.930 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2023-11-01T04:16:48.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.930 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:178 3.9s0 3:94ead3c8:::smithi19039307-33:head [read 0~540672,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:48.930 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.939732+0000", 2023-11-01T04:16:48.930 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.987786902, 2023-11-01T04:16:48.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30648703300000002, 2023-11-01T04:16:48.931 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.931 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:48.931 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:48.931 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:48.932 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:48.932 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 178 2023-11-01T04:16:48.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.932 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939732+0000", 2023-11-01T04:16:48.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.934 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939732+0000", 2023-11-01T04:16:48.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.934 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.935 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939735+0000", 2023-11-01T04:16:48.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6199999999999999e-06 2023-11-01T04:16:48.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.935 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.935 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.936 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939748+0000", 2023-11-01T04:16:48.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2704999999999999e-05 2023-11-01T04:16:48.936 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.936 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939780+0000", 2023-11-01T04:16:48.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2478000000000003e-05 2023-11-01T04:16:48.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939796+0000", 2023-11-01T04:16:48.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5492999999999999e-05 2023-11-01T04:16:48.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939948+0000", 2023-11-01T04:16:48.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015278000000000001 2023-11-01T04:16:48.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:48.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.940137+0000", 2023-11-01T04:16:48.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018902400000000001 2023-11-01T04:16:48.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.940 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.246219+0000", 2023-11-01T04:16:48.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30608193299999997 2023-11-01T04:16:48.941 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.941 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.941 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.942 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:179 3.9s0 3:94ead3c8:::smithi19039307-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:48.942 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.939832+0000", 2023-11-01T04:16:48.942 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.987687339000001, 2023-11-01T04:16:48.942 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30729425199999999, 2023-11-01T04:16:48.942 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.943 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:48.943 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:48.943 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:48.943 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:48.943 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 179 2023-11-01T04:16:48.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.944 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939832+0000", 2023-11-01T04:16:48.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939832+0000", 2023-11-01T04:16:48.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.946 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.946 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939833+0000", 2023-11-01T04:16:48.946 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2240000000000001e-06 2023-11-01T04:16:48.946 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.947 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.947 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939843+0000", 2023-11-01T04:16:48.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5659999999999996e-06 2023-11-01T04:16:48.947 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.948 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939864+0000", 2023-11-01T04:16:48.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1098e-05 2023-11-01T04:16:48.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939882+0000", 2023-11-01T04:16:48.949 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8031999999999999e-05 2023-11-01T04:16:48.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.949 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.940794+0000", 2023-11-01T04:16:48.950 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00091202099999999995 2023-11-01T04:16:48.950 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:48.951 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.941019+0000", 2023-11-01T04:16:48.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022529900000000001 2023-11-01T04:16:48.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.951 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.951 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.952 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.247126+0000", 2023-11-01T04:16:48.952 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30610701200000001 2023-11-01T04:16:48.952 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.952 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.952 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.952 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.953 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:180 3.9s0 3:94ead3c8:::smithi19039307-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:48.953 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.939918+0000", 2023-11-01T04:16:48.953 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.987601728000001, 2023-11-01T04:16:48.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30792408900000001, 2023-11-01T04:16:48.953 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.954 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:48.954 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:48.954 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:48.954 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:48.954 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 180 2023-11-01T04:16:48.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.955 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939918+0000", 2023-11-01T04:16:48.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.956 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.956 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939918+0000", 2023-11-01T04:16:48.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939919+0000", 2023-11-01T04:16:48.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.187e-06 2023-11-01T04:16:48.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939927+0000", 2023-11-01T04:16:48.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0059999999999992e-06 2023-11-01T04:16:48.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939951+0000", 2023-11-01T04:16:48.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3759000000000001e-05 2023-11-01T04:16:48.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.960 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.960 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.939969+0000", 2023-11-01T04:16:48.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8402000000000001e-05 2023-11-01T04:16:48.960 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.943066+0000", 2023-11-01T04:16:48.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003097266 2023-11-01T04:16:48.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:48.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.943281+0000", 2023-11-01T04:16:48.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021503900000000001 2023-11-01T04:16:48.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.247842+0000", 2023-11-01T04:16:48.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30456042999999999 2023-11-01T04:16:48.963 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.963 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.963 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.964 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.9s0 36/31 ECSubReadReply(tid=10, attrs_read=0))", 2023-11-01T04:16:48.964 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.942329+0000", 2023-11-01T04:16:48.964 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.985190224, 2023-11-01T04:16:48.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30392635899999998, 2023-11-01T04:16:48.965 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.965 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:48.965 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.942329+0000", 2023-11-01T04:16:48.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.942329+0000", 2023-11-01T04:16:48.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.942331+0000", 2023-11-01T04:16:48.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.584e-06 2023-11-01T04:16:48.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.942856+0000", 2023-11-01T04:16:48.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000525856 2023-11-01T04:16:48.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.942886+0000", 2023-11-01T04:16:48.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9471e-05 2023-11-01T04:16:48.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.970 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.970 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.970 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.042979+0000", 2023-11-01T04:16:48.970 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100092548 2023-11-01T04:16:48.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.245459+0000", 2023-11-01T04:16:48.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20248086400000001 2023-11-01T04:16:48.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.246255+0000", 2023-11-01T04:16:48.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00079603599999999997 2023-11-01T04:16:48.972 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.972 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.973 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.973 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:181 3.9s0 3:959ff18f:::smithi19039307-32:head [append 0~524288] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:48.973 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.957492+0000", 2023-11-01T04:16:48.973 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.970027772000002, 2023-11-01T04:16:48.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.62014424999999995, 2023-11-01T04:16:48.974 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.974 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:48.974 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:48.974 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:48.974 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:48.975 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 181 2023-11-01T04:16:48.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.975 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.957492+0000", 2023-11-01T04:16:48.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.957492+0000", 2023-11-01T04:16:48.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.957494+0000", 2023-11-01T04:16:48.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4430000000000002e-06 2023-11-01T04:16:48.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.958130+0000", 2023-11-01T04:16:48.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00063580600000000004 2023-11-01T04:16:48.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.958162+0000", 2023-11-01T04:16:48.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2670999999999999e-05 2023-11-01T04:16:48.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.958179+0000", 2023-11-01T04:16:48.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6631999999999999e-05 2023-11-01T04:16:48.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.981 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.981 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.247961+0000", 2023-11-01T04:16:48.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28978161600000002 2023-11-01T04:16:48.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:48.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.248112+0000", 2023-11-01T04:16:48.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000151781 2023-11-01T04:16:48.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:48.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.250243+0000", 2023-11-01T04:16:48.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021303839999999999 2023-11-01T04:16:48.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:48.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.258223+0000", 2023-11-01T04:16:48.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0079804449999999992 2023-11-01T04:16:48.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:48.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.577520+0000", 2023-11-01T04:16:48.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31929720700000003 2023-11-01T04:16:48.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.985 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.577636+0000", 2023-11-01T04:16:48.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000115265 2023-11-01T04:16:48.986 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.986 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.986 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.986 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:182 3.9s0 3:959ff18f:::smithi19039307-32:head [append 0~376832] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:48.987 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.963714+0000", 2023-11-01T04:16:48.987 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.963805595, 2023-11-01T04:16:48.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.61422031099999996, 2023-11-01T04:16:48.987 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:48.987 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:48.988 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:48.988 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:48.988 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:48.988 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 182 2023-11-01T04:16:48.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.988 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:48.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.989 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:48.989 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963714+0000", 2023-11-01T04:16:48.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.990 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:48.990 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963714+0000", 2023-11-01T04:16:48.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:48.990 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:48.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963716+0000", 2023-11-01T04:16:48.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3099999999999999e-06 2023-11-01T04:16:48.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:48.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964289+0000", 2023-11-01T04:16:48.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00057335299999999997 2023-11-01T04:16:48.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:48.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964323+0000", 2023-11-01T04:16:48.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3268000000000001e-05 2023-11-01T04:16:48.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:48.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964339+0000", 2023-11-01T04:16:48.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6782e-05 2023-11-01T04:16:48.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:48.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.252517+0000", 2023-11-01T04:16:48.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.28817765499999998 2023-11-01T04:16:48.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:48.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.252667+0000", 2023-11-01T04:16:48.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015000300000000001 2023-11-01T04:16:48.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:48.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.254483+0000", 2023-11-01T04:16:48.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018163249999999999 2023-11-01T04:16:48.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:48.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.258226+0000", 2023-11-01T04:16:48.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037425750000000002 2023-11-01T04:16:48.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:48.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.577851+0000", 2023-11-01T04:16:48.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31962504200000003 2023-11-01T04:16:48.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:48.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:48.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:48.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.577934+0000", 2023-11-01T04:16:48.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2997999999999994e-05 2023-11-01T04:16:48.999 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.999 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:48.999 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:48.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.000 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:183 3.9s0 3:959ff18f:::smithi19039307-32:head [setxattr _header (56) in=7b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:49.000 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.964367+0000", 2023-11-01T04:16:49.000 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.963152431000001, 2023-11-01T04:16:49.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.61393845700000005, 2023-11-01T04:16:49.000 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.001 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:49.001 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.001 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.001 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.001 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 183 2023-11-01T04:16:49.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.002 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964367+0000", 2023-11-01T04:16:49.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964367+0000", 2023-11-01T04:16:49.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964368+0000", 2023-11-01T04:16:49.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3039999999999999e-06 2023-11-01T04:16:49.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964380+0000", 2023-11-01T04:16:49.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1905999999999999e-05 2023-11-01T04:16:49.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964392+0000", 2023-11-01T04:16:49.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1769e-05 2023-11-01T04:16:49.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964401+0000", 2023-11-01T04:16:49.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9260000000000006e-06 2023-11-01T04:16:49.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.255768+0000", 2023-11-01T04:16:49.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.291367502 2023-11-01T04:16:49.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.256021+0000", 2023-11-01T04:16:49.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025249400000000003 2023-11-01T04:16:49.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:49.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.257165+0000", 2023-11-01T04:16:49.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00114455 2023-11-01T04:16:49.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:49.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.358396+0000", 2023-11-01T04:16:49.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101230256 2023-11-01T04:16:49.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:49.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.578140+0000", 2023-11-01T04:16:49.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21974475700000001 2023-11-01T04:16:49.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.578305+0000", 2023-11-01T04:16:49.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000164993 2023-11-01T04:16:49.012 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.012 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.013 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.013 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:184 3.9s0 3:959ff18f:::smithi19039307-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:49.013 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.964414+0000", 2023-11-01T04:16:49.014 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.9631057, 2023-11-01T04:16:49.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.72156181200000002, 2023-11-01T04:16:49.014 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.014 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:49.014 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.014 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.015 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.015 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 184 2023-11-01T04:16:49.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.015 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964414+0000", 2023-11-01T04:16:49.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964414+0000", 2023-11-01T04:16:49.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964414+0000", 2023-11-01T04:16:49.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2600000000000001e-07 2023-11-01T04:16:49.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964418+0000", 2023-11-01T04:16:49.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5899999999999999e-06 2023-11-01T04:16:49.080 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.081 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.081 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.081 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964429+0000", 2023-11-01T04:16:49.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0616e-05 2023-11-01T04:16:49.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.082 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964437+0000", 2023-11-01T04:16:49.082 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9470000000000003e-06 2023-11-01T04:16:49.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.083 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.259490+0000", 2023-11-01T04:16:49.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29505374699999998 2023-11-01T04:16:49.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for rw locks", 2023-11-01T04:16:49.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.259647+0000", 2023-11-01T04:16:49.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015683599999999999 2023-11-01T04:16:49.084 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.085 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.579080+0000", 2023-11-01T04:16:49.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31943243399999999 2023-11-01T04:16:49.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.579233+0000", 2023-11-01T04:16:49.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000153621 2023-11-01T04:16:49.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.685975+0000", 2023-11-01T04:16:49.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.106742195 2023-11-01T04:16:49.087 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.087 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.087 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.088 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:406 3.9s0 3:94ead3c8:::smithi19039307-33:head [copy-from ver 4] snapc 12=[12,11,10,f,e,d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2023-11-01T04:16:49.088 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:33.107132+0000", 2023-11-01T04:16:49.088 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.820387586000001, 2023-11-01T04:16:49.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.61613177100000005, 2023-11-01T04:16:49.088 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.089 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:49.089 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.089 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.089 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.089 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 406 2023-11-01T04:16:49.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.090 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.107132+0000", 2023-11-01T04:16:49.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.107132+0000", 2023-11-01T04:16:49.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.107136+0000", 2023-11-01T04:16:49.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2300000000000002e-06 2023-11-01T04:16:49.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.107148+0000", 2023-11-01T04:16:49.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2316e-05 2023-11-01T04:16:49.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.107182+0000", 2023-11-01T04:16:49.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3419e-05 2023-11-01T04:16:49.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.107200+0000", 2023-11-01T04:16:49.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7875000000000001e-05 2023-11-01T04:16:49.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.107338+0000", 2023-11-01T04:16:49.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000138808 2023-11-01T04:16:49.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2023-11-01T04:16:49.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.107351+0000", 2023-11-01T04:16:49.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3016e-05 2023-11-01T04:16:49.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.114702+0000", 2023-11-01T04:16:49.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0073507540000000001 2023-11-01T04:16:49.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.114877+0000", 2023-11-01T04:16:49.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000175243 2023-11-01T04:16:49.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:49.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.419395+0000", 2023-11-01T04:16:49.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30451812700000003 2023-11-01T04:16:49.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:49.100 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.522020+0000", 2023-11-01T04:16:49.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10262471200000001 2023-11-01T04:16:49.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:49.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.723140+0000", 2023-11-01T04:16:49.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.201119989 2023-11-01T04:16:49.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.723263+0000", 2023-11-01T04:16:49.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000123282 2023-11-01T04:16:49.102 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.102 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.102 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.103 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:407 3.9s0 3:94ead3c8:::smithi19039307-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2023-11-01T04:16:49.103 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:33.107237+0000", 2023-11-01T04:16:49.103 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.820282037, 2023-11-01T04:16:49.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.314343447, 2023-11-01T04:16:49.103 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.104 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:49.104 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.104 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.104 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.104 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 407 2023-11-01T04:16:49.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.105 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.107237+0000", 2023-11-01T04:16:49.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.107237+0000", 2023-11-01T04:16:49.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.107264+0000", 2023-11-01T04:16:49.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6732000000000001e-05 2023-11-01T04:16:49.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.107286+0000", 2023-11-01T04:16:49.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1982999999999999e-05 2023-11-01T04:16:49.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.107302+0000", 2023-11-01T04:16:49.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6206e-05 2023-11-01T04:16:49.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.107313+0000", 2023-11-01T04:16:49.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0463e-05 2023-11-01T04:16:49.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.111329+0000", 2023-11-01T04:16:49.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0040159729999999999 2023-11-01T04:16:49.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:49.112 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.111348+0000", 2023-11-01T04:16:49.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9825000000000001e-05 2023-11-01T04:16:49.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.112 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.115174+0000", 2023-11-01T04:16:49.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003825448 2023-11-01T04:16:49.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2023-11-01T04:16:49.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.115256+0000", 2023-11-01T04:16:49.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2479999999999996e-05 2023-11-01T04:16:49.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.421257+0000", 2023-11-01T04:16:49.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30600055300000001 2023-11-01T04:16:49.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.421423+0000", 2023-11-01T04:16:49.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016603000000000001 2023-11-01T04:16:49.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.421581+0000", 2023-11-01T04:16:49.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015775399999999999 2023-11-01T04:16:49.116 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.117 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.117 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.117 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:408 3.9s0 3:959ff18f:::smithi19039307-32:head [create,setxattr h^blbcaNtSmmsQtZPdaRtPkhOOgN_Vi\\f`NtsZ[V (561) in=40b,setxattr T (17332) in=1b,setxattr UVoaZteYfmpthpcfl[gebaR\\TdqcNkhPRRdrgsX] (279) in=40b,setxattr bTPgfXb\\YXtXkQag\\Qh`hlXr[posd]]cje]^eebl (1121) in=40b,setxattr hm^gakWSOOOfNOTph^tma]iaNsaqfjqZYb\\seZNW (3297) in=40b,setxattr YiNn[]m_q`YlQlaV^mRn^ (17332) in=21b,setxattr \\tllpkXThmWXSkOopogrdtjY]f^a^sem_krafjfi (947) in=40b,setxattr RXiRZRQVSaUTaZmQT^bfqZhS^mZsqadaPdW]ijm` (1963) in=40b,setxattr WTmsTZhNpU`tZY\\^aROaNdmtYsjjsgWWboehPOP` (19508) in=40b,setxattr UZmWZeWaoiX]jmbn][UadnNkcZc_hdcjidtNsRYo (1999) in=40b,setxattr NSebjRYYQ`SbaatVjZWNab`cenkljRq`^]Xr\\jRm (11788) in=40b,setxattr ip`daUaop[rdbNPo\\^c[Wcelsl]a][RScQZ[kfa^ (18524) in=40b,setxattr qU_qcNcQpNkegs_Od`jieTXX[mto]lsR^jTcYprR (593) in=40b,setxattr dboVg`srq[bReagdR]fX\\Pe`Warrp]\\[XpSipTN[ (1683) in=40b,setxattr ZdTaVj[ScjsOiQ]a_]R[RQWoYohnjd`fi[dYmfs` (840) in=40b,setxattr N_[blmbpstWhtUgW]ZjZ[\\O_erqcnQN]R_V^U\\RT (1015) in=40b,setxattr jXbP_mNsadnq`UVfmc^lfZUs\\r_[ZnrahqTUk^Zp (383) in=40b,setxattr nmemd\\ipehj\\q^UVY_YtSrhZjcabNXhlai_noQSO (807) in=40b,setxattr Rbc\\cahhlcrckUioieUbQnrseNkZkdPf\\rUmNb]S (384) in=40b,setxattr ZiOkXqg^PobTdZgrtZ]]l^Rap_tfSQoUQqaefj]] (69) in=40b] snapc 12=[12,11,10,f,e,d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2023-11-01T04:16:49.117 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:33.113949+0000", 2023-11-01T04:16:49.118 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 15.813570134000001, 2023-11-01T04:16:49.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30178434900000001, 2023-11-01T04:16:49.118 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.118 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:49.118 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.118 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.119 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.119 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 408 2023-11-01T04:16:49.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.119 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.113949+0000", 2023-11-01T04:16:49.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.113949+0000", 2023-11-01T04:16:49.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.113952+0000", 2023-11-01T04:16:49.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1300000000000001e-06 2023-11-01T04:16:49.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.114133+0000", 2023-11-01T04:16:49.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018090000000000001 2023-11-01T04:16:49.123 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.123 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.123 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.114166+0000", 2023-11-01T04:16:49.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3297999999999999e-05 2023-11-01T04:16:49.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.114182+0000", 2023-11-01T04:16:49.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5536000000000002e-05 2023-11-01T04:16:49.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.207544+0000", 2023-11-01T04:16:49.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.093362111999999997 2023-11-01T04:16:49.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.207741+0000", 2023-11-01T04:16:49.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019685899999999999 2023-11-01T04:16:49.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:49.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.209848+0000", 2023-11-01T04:16:49.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021072439999999999 2023-11-01T04:16:49.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:49.128 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.216745+0000", 2023-11-01T04:16:49.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068969690000000002 2023-11-01T04:16:49.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:49.129 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.415517+0000", 2023-11-01T04:16:49.129 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.19877164 2023-11-01T04:16:49.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.129 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.130 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:33.415733+0000", 2023-11-01T04:16:49.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021666099999999999 2023-11-01T04:16:49.130 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.130 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.130 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.130 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.131 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:482 3.9s0 3:959ff18f:::smithi19039307-32:head [read 0~1425408,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2023-11-01T04:16:49.131 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:42.289701+0000", 2023-11-01T04:16:49.131 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.6378184359999999, 2023-11-01T04:16:49.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.32590008399999998, 2023-11-01T04:16:49.131 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.132 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:49.132 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.132 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.132 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.132 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 482 2023-11-01T04:16:49.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.133 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.289701+0000", 2023-11-01T04:16:49.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.289701+0000", 2023-11-01T04:16:49.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.135 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.289704+0000", 2023-11-01T04:16:49.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.45e-06 2023-11-01T04:16:49.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.289736+0000", 2023-11-01T04:16:49.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1238999999999997e-05 2023-11-01T04:16:49.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.289759+0000", 2023-11-01T04:16:49.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3768999999999999e-05 2023-11-01T04:16:49.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.289796+0000", 2023-11-01T04:16:49.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6872000000000002e-05 2023-11-01T04:16:49.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.138 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.138 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.291115+0000", 2023-11-01T04:16:49.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013185499999999999 2023-11-01T04:16:49.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.291366+0000", 2023-11-01T04:16:49.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025126800000000002 2023-11-01T04:16:49.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.615601+0000", 2023-11-01T04:16:49.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.32423493599999997 2023-11-01T04:16:49.141 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.141 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.141 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.142 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:483 3.9s0 3:959ff18f:::smithi19039307-32:head [read 0~1425408,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2023-11-01T04:16:49.142 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:42.289868+0000", 2023-11-01T04:16:49.142 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.637651355, 2023-11-01T04:16:49.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.83200423300000004, 2023-11-01T04:16:49.142 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.142 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:49.143 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.143 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.143 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.143 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 483 2023-11-01T04:16:49.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.143 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.144 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.144 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.289868+0000", 2023-11-01T04:16:49.144 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.145 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.145 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.289868+0000", 2023-11-01T04:16:49.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.145 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.145 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.289869+0000", 2023-11-01T04:16:49.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.486e-06 2023-11-01T04:16:49.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.147 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.147 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.289878+0000", 2023-11-01T04:16:49.147 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7199999999999995e-06 2023-11-01T04:16:49.147 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.147 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.147 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.289899+0000", 2023-11-01T04:16:49.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0962999999999999e-05 2023-11-01T04:16:49.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.289912+0000", 2023-11-01T04:16:49.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2680000000000001e-05 2023-11-01T04:16:49.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.291992+0000", 2023-11-01T04:16:49.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020801309999999998 2023-11-01T04:16:49.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.292125+0000", 2023-11-01T04:16:49.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000133399 2023-11-01T04:16:49.151 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.151 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:43.121872+0000", 2023-11-01T04:16:49.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.82974685400000003 2023-11-01T04:16:49.151 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.152 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.152 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.152 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:484 3.9s0 3:9c07f031:::smithi19039307-43:head [append 0~655360] snapc 1b=[1b,19,18,15,12,11,b,4] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2023-11-01T04:16:49.152 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:42.289946+0000", 2023-11-01T04:16:49.153 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.6375731880000002, 2023-11-01T04:16:49.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.42629635300000002, 2023-11-01T04:16:49.153 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.153 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:49.153 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.154 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.154 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.154 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 484 2023-11-01T04:16:49.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.154 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.289946+0000", 2023-11-01T04:16:49.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.289946+0000", 2023-11-01T04:16:49.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.289948+0000", 2023-11-01T04:16:49.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.53e-06 2023-11-01T04:16:49.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.290905+0000", 2023-11-01T04:16:49.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00095748899999999995 2023-11-01T04:16:49.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.290936+0000", 2023-11-01T04:16:49.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1192000000000002e-05 2023-11-01T04:16:49.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.159 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.290963+0000", 2023-11-01T04:16:49.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7109000000000001e-05 2023-11-01T04:16:49.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.160 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.160 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.160 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.396380+0000", 2023-11-01T04:16:49.160 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.105416605 2023-11-01T04:16:49.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.396661+0000", 2023-11-01T04:16:49.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028126699999999998 2023-11-01T04:16:49.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.162 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.162 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:49.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.400165+0000", 2023-11-01T04:16:49.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0035033719999999998 2023-11-01T04:16:49.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:49.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.612582+0000", 2023-11-01T04:16:49.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21241744400000001 2023-11-01T04:16:49.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:49.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.716122+0000", 2023-11-01T04:16:49.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10353978899999999 2023-11-01T04:16:49.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.716242+0000", 2023-11-01T04:16:49.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000120556 2023-11-01T04:16:49.165 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.165 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.165 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.166 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:485 3.9s0 3:9c07f031:::smithi19039307-43:head [append 0~638976] snapc 1b=[1b,19,18,15,12,11,b,4] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2023-11-01T04:16:49.166 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:42.290997+0000", 2023-11-01T04:16:49.166 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.636522212, 2023-11-01T04:16:49.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.62611375800000002, 2023-11-01T04:16:49.167 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.167 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:49.167 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.167 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.167 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.167 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 485 2023-11-01T04:16:49.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.168 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.290997+0000", 2023-11-01T04:16:49.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.290997+0000", 2023-11-01T04:16:49.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.290999+0000", 2023-11-01T04:16:49.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5460000000000001e-06 2023-11-01T04:16:49.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.291965+0000", 2023-11-01T04:16:49.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00096665800000000001 2023-11-01T04:16:49.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.291991+0000", 2023-11-01T04:16:49.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5867000000000002e-05 2023-11-01T04:16:49.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.392098+0000", 2023-11-01T04:16:49.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100106608 2023-11-01T04:16:49.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.505522+0000", 2023-11-01T04:16:49.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.11342387600000001 2023-11-01T04:16:49.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.505684+0000", 2023-11-01T04:16:49.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016205500000000001 2023-11-01T04:16:49.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:49.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.507430+0000", 2023-11-01T04:16:49.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00174647 2023-11-01T04:16:49.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:49.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.612752+0000", 2023-11-01T04:16:49.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.105321425 2023-11-01T04:16:49.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:49.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.916935+0000", 2023-11-01T04:16:49.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30418373900000001 2023-11-01T04:16:49.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.917111+0000", 2023-11-01T04:16:49.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017551399999999999 2023-11-01T04:16:49.182 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.183 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.183 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.184 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:486 3.9s0 3:9c07f031:::smithi19039307-43:head [append 0~131072] snapc 1b=[1b,19,18,15,12,11,b,4] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2023-11-01T04:16:49.184 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:42.392132+0000", 2023-11-01T04:16:49.185 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.5353876609999997, 2023-11-01T04:16:49.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.62550418200000002, 2023-11-01T04:16:49.185 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.186 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:49.186 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.187 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.187 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.187 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 486 2023-11-01T04:16:49.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.188 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.392132+0000", 2023-11-01T04:16:49.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.392132+0000", 2023-11-01T04:16:49.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.392133+0000", 2023-11-01T04:16:49.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8500000000000001e-06 2023-11-01T04:16:49.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.194 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.194 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.194 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.392222+0000", 2023-11-01T04:16:49.195 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8694000000000002e-05 2023-11-01T04:16:49.195 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.195 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.196 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.196 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.392246+0000", 2023-11-01T04:16:49.196 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4226000000000001e-05 2023-11-01T04:16:49.197 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.197 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.198 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.198 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.492366+0000", 2023-11-01T04:16:49.198 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100119308 2023-11-01T04:16:49.199 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.199 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.199 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.200 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.609271+0000", 2023-11-01T04:16:49.200 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.116904991 2023-11-01T04:16:49.200 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.201 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.201 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.201 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.609574+0000", 2023-11-01T04:16:49.202 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030363100000000001 2023-11-01T04:16:49.202 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.203 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.203 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:49.203 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.611203+0000", 2023-11-01T04:16:49.204 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001628336 2023-11-01T04:16:49.204 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.204 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.205 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:49.205 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.615656+0000", 2023-11-01T04:16:49.205 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044538080000000001 2023-11-01T04:16:49.206 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.206 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.206 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:49.207 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:43.017468+0000", 2023-11-01T04:16:49.207 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40181193700000001 2023-11-01T04:16:49.208 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.208 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.208 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.209 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:43.017636+0000", 2023-11-01T04:16:49.209 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016740099999999999 2023-11-01T04:16:49.209 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.210 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.210 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.211 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.211 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.211 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:487 3.9s0 3:9c07f031:::smithi19039307-43:head [setxattr _header (57) in=7b] snapc 1b=[1b,19,18,15,12,11,b,4] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2023-11-01T04:16:49.212 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:42.492406+0000", 2023-11-01T04:16:49.212 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.435113168, 2023-11-01T04:16:49.212 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.62577764599999997, 2023-11-01T04:16:49.213 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.213 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:49.213 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.214 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.214 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.215 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 487 2023-11-01T04:16:49.215 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.215 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.216 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.216 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.216 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.492406+0000", 2023-11-01T04:16:49.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.492406+0000", 2023-11-01T04:16:49.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.492408+0000", 2023-11-01T04:16:49.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.34e-06 2023-11-01T04:16:49.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.492421+0000", 2023-11-01T04:16:49.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2593e-05 2023-11-01T04:16:49.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.492456+0000", 2023-11-01T04:16:49.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4822e-05 2023-11-01T04:16:49.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.492500+0000", 2023-11-01T04:16:49.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3671999999999998e-05 2023-11-01T04:16:49.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.615870+0000", 2023-11-01T04:16:49.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.123370513 2023-11-01T04:16:49.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.616157+0000", 2023-11-01T04:16:49.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028646100000000001 2023-11-01T04:16:49.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:49.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.617464+0000", 2023-11-01T04:16:49.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001307424 2023-11-01T04:16:49.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:49.231 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.715947+0000", 2023-11-01T04:16:49.231 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.098482880999999994 2023-11-01T04:16:49.232 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.232 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.232 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:49.233 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:43.117991+0000", 2023-11-01T04:16:49.233 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40204426599999998 2023-11-01T04:16:49.233 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.234 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.234 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.234 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:43.118184+0000", 2023-11-01T04:16:49.235 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019267399999999999 2023-11-01T04:16:49.235 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.235 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.235 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.236 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.236 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.236 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:488 3.9s0 3:9c07f031:::smithi19039307-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2023-11-01T04:16:49.236 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:42.492549+0000", 2023-11-01T04:16:49.237 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.4349698589999997, 2023-11-01T04:16:49.237 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.83278839400000004, 2023-11-01T04:16:49.237 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.238 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:49.238 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.238 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.239 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.239 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 488 2023-11-01T04:16:49.239 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.240 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.240 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.240 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.241 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.492549+0000", 2023-11-01T04:16:49.241 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.242 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.242 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.243 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.492549+0000", 2023-11-01T04:16:49.243 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.243 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.244 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.244 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.244 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.492551+0000", 2023-11-01T04:16:49.245 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.606e-06 2023-11-01T04:16:49.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.245 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.245 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.246 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.492559+0000", 2023-11-01T04:16:49.246 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1200000000000002e-06 2023-11-01T04:16:49.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.247 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.247 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.247 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.492582+0000", 2023-11-01T04:16:49.248 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3059000000000001e-05 2023-11-01T04:16:49.248 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.248 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.249 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.249 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.492599+0000", 2023-11-01T04:16:49.249 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6707000000000001e-05 2023-11-01T04:16:49.250 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.250 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.250 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.251 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.619429+0000", 2023-11-01T04:16:49.251 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.12683035300000001 2023-11-01T04:16:49.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.252 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.252 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for rw locks", 2023-11-01T04:16:49.252 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.619656+0000", 2023-11-01T04:16:49.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000226686 2023-11-01T04:16:49.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.253 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.253 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.253 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:43.119291+0000", 2023-11-01T04:16:49.253 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.49963538400000002 2023-11-01T04:16:49.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.254 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.254 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.254 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:43.119451+0000", 2023-11-01T04:16:49.254 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000159269 2023-11-01T04:16:49.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.255 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.255 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.255 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:43.325338+0000", 2023-11-01T04:16:49.255 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20588720999999999 2023-11-01T04:16:49.256 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.256 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.256 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.256 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.256 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.256 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.9s0 62/31 ECSubReadReply(tid=73, attrs_read=0))", 2023-11-01T04:16:49.257 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:42.621548+0000", 2023-11-01T04:16:49.257 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.3059712399999999, 2023-11-01T04:16:49.257 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.50040231199999996, 2023-11-01T04:16:49.257 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.257 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:49.257 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.258 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.258 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.258 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.621548+0000", 2023-11-01T04:16:49.258 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.258 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.259 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.259 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.621548+0000", 2023-11-01T04:16:49.259 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.259 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.260 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.260 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.621549+0000", 2023-11-01T04:16:49.260 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.043e-06 2023-11-01T04:16:49.260 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.260 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.261 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.261 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.621811+0000", 2023-11-01T04:16:49.261 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000262305 2023-11-01T04:16:49.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.261 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.262 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.262 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.621835+0000", 2023-11-01T04:16:49.262 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3728999999999999e-05 2023-11-01T04:16:49.262 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.262 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.263 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.621851+0000", 2023-11-01T04:16:49.263 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6322999999999999e-05 2023-11-01T04:16:49.263 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.263 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.263 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:43.120430+0000", 2023-11-01T04:16:49.264 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.49857897699999998 2023-11-01T04:16:49.264 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.264 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.264 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.264 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:43.121950+0000", 2023-11-01T04:16:49.265 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001519935 2023-11-01T04:16:49.265 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.265 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.265 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.265 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.265 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.266 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 62/31 ECSubWriteReply(tid=77, last_complete=62'66, committed=1, applied=1))", 2023-11-01T04:16:49.266 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:42.813840+0000", 2023-11-01T04:16:49.266 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.1136792150000003, 2023-11-01T04:16:49.266 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.305326712, 2023-11-01T04:16:49.266 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.267 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:49.267 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.267 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.267 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.267 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.813840+0000", 2023-11-01T04:16:49.267 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.268 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.268 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.268 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.268 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.813840+0000", 2023-11-01T04:16:49.268 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.269 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.269 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.813842+0000", 2023-11-01T04:16:49.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7570000000000001e-06 2023-11-01T04:16:49.269 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.270 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.270 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.813851+0000", 2023-11-01T04:16:49.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0159999999999997e-06 2023-11-01T04:16:49.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.813880+0000", 2023-11-01T04:16:49.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9252999999999999e-05 2023-11-01T04:16:49.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:42.813894+0000", 2023-11-01T04:16:49.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4209000000000001e-05 2023-11-01T04:16:49.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:43.117865+0000", 2023-11-01T04:16:49.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30397048199999999 2023-11-01T04:16:49.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:43.119167+0000", 2023-11-01T04:16:49.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001301995 2023-11-01T04:16:49.274 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.274 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.274 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.274 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.275 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.275 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:49.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-01T04:16:49.277 INFO:tasks.rados.rados.0.smithi190.stdout:226: done (1 left) 2023-11-01T04:16:49.277 INFO:tasks.rados.rados.0.smithi190.stdout:227: delete oid 30 current snap is 21 2023-11-01T04:16:49.278 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:49.275+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:49.292 INFO:tasks.rados.rados.0.smithi190.stdout:227: done (1 left) 2023-11-01T04:16:49.292 INFO:tasks.rados.rados.0.smithi190.stdout:228: copy_from oid 41 from oid 11 current snap is 21 2023-11-01T04:16:49.292 INFO:tasks.rados.rados.0.smithi190.stdout:229: delete oid 29 current snap is 21 2023-11-01T04:16:49.377 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:49.375+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:49.393 INFO:tasks.rados.rados.0.smithi190.stdout:214: finishing write tid 5 to smithi19039307-19 2023-11-01T04:16:49.393 INFO:tasks.rados.rados.0.smithi190.stdout:214: oid 19 updating version 0 to 30 2023-11-01T04:16:49.393 INFO:tasks.rados.rados.0.smithi190.stdout:214: oid 19 version 30 is already newer than 29 2023-11-01T04:16:49.393 INFO:tasks.rados.rados.0.smithi190.stdout:214: oid 19 version 30 is already newer than 28 2023-11-01T04:16:49.394 INFO:tasks.rados.rados.0.smithi190.stdout:214: oid 19 version 30 is already newer than 27 2023-11-01T04:16:49.394 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 19 v 30 (ObjNum 89 snap 20 seq_num 89) dirty exists 2023-11-01T04:16:49.394 INFO:tasks.rados.rados.0.smithi190.stdout:214: left oid 19 (ObjNum 89 snap 20 seq_num 89) 2023-11-01T04:16:49.407 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:49.408 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:49.408 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2023-11-01T04:16:49.408 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:49.422 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-01T04:16:49.477 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:16:49.496 INFO:tasks.rados.rados.0.smithi190.stdout:214: done (2 left) 2023-11-01T04:16:49.496 INFO:tasks.rados.rados.0.smithi190.stdout:229: done (1 left) 2023-11-01T04:16:49.496 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 14 current snap is 21 2023-11-01T04:16:49.497 INFO:tasks.rados.rados.0.smithi190.stdout:230: seq_num 94 ranges {0=425984,425984=786432,1212416=376832} 2023-11-01T04:16:49.499 INFO:tasks.rados.rados.0.smithi190.stdout:230: writing smithi19039307-14 from 0 to 425984 tid 1 2023-11-01T04:16:49.506 INFO:tasks.rados.rados.0.smithi190.stdout:230: writing smithi19039307-14 from 425984 to 1212416 tid 2 2023-11-01T04:16:49.510 INFO:tasks.rados.rados.0.smithi190.stdout:230: writing smithi19039307-14 from 1212416 to 1589248 tid 3 2023-11-01T04:16:49.510 INFO:tasks.rados.rados.0.smithi190.stdout:231: read oid 2 snap -1 2023-11-01T04:16:49.510 INFO:tasks.rados.rados.0.smithi190.stdout:231: expect (ObjNum 33 snap 0 seq_num 33) 2023-11-01T04:16:49.510 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 24 current snap is 21 2023-11-01T04:16:49.511 INFO:tasks.rados.rados.0.smithi190.stdout:232: seq_num 95 ranges {2244608=442368,2686976=540672,3227648=622592} 2023-11-01T04:16:49.530 INFO:tasks.rados.rados.0.smithi190.stdout:232: writing smithi19039307-24 from 2244608 to 2686976 tid 1 2023-11-01T04:16:49.534 INFO:tasks.rados.rados.0.smithi190.stdout:232: writing smithi19039307-24 from 2686976 to 3227648 tid 2 2023-11-01T04:16:49.538 INFO:tasks.rados.rados.0.smithi190.stdout:232: writing smithi19039307-24 from 3227648 to 3850240 tid 3 2023-11-01T04:16:49.539 INFO:tasks.rados.rados.0.smithi190.stdout:233: delete oid 45 current snap is 21 2023-11-01T04:16:49.580 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:49.579+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:49.592 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:49.593 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:49.593 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2023-11-01T04:16:49.593 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2023-11-01T04:16:49.593 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:49.603 INFO:tasks.rados.rados.0.smithi190.stdout:228: finishing copy_from racing read to smithi19039307-41 2023-11-01T04:16:49.607 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-01T04:16:49.681 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:49.681+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:49.705 INFO:tasks.rados.rados.0.smithi190.stdout:228: finishing copy_from to smithi19039307-41 2023-11-01T04:16:49.705 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 41 v 4 (ObjNum 10 snap 0 seq_num 10) dirty exists 2023-11-01T04:16:49.712 INFO:tasks.rados.rados.0.smithi190.stdout:230: finishing write tid 1 to smithi19039307-14 2023-11-01T04:16:49.777 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:49.777 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2023-11-01T04:16:49.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2023-11-01T04:16:49.778 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2023-11-01T04:16:49.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.778 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:185 3.es0 3:723efae4:::smithi19039307-36:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:49.778 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.963781+0000", 2023-11-01T04:16:49.779 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.812252346999998, 2023-11-01T04:16:49.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31012846799999999, 2023-11-01T04:16:49.779 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.779 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:49.779 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.780 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.780 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.780 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185 2023-11-01T04:16:49.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.780 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.781 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.781 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963781+0000", 2023-11-01T04:16:49.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.782 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.782 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963781+0000", 2023-11-01T04:16:49.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.783 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963785+0000", 2023-11-01T04:16:49.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.563e-06 2023-11-01T04:16:49.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.783 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963797+0000", 2023-11-01T04:16:49.784 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2201999999999999e-05 2023-11-01T04:16:49.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963839+0000", 2023-11-01T04:16:49.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2181000000000002e-05 2023-11-01T04:16:49.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.786 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963864+0000", 2023-11-01T04:16:49.786 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4419000000000001e-05 2023-11-01T04:16:49.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.786 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.786 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964036+0000", 2023-11-01T04:16:49.787 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017209500000000001 2023-11-01T04:16:49.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.787 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964291+0000", 2023-11-01T04:16:49.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025550699999999998 2023-11-01T04:16:49.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:49.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966403+0000", 2023-11-01T04:16:49.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021113500000000001 2023-11-01T04:16:49.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:49.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.072479+0000", 2023-11-01T04:16:49.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10607620800000001 2023-11-01T04:16:49.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.790 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:49.790 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.273808+0000", 2023-11-01T04:16:49.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20132876299999999 2023-11-01T04:16:49.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.791 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.273910+0000", 2023-11-01T04:16:49.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010218 2023-11-01T04:16:49.792 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.792 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.792 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.792 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:186 3.es0 3:723efae4:::smithi19039307-36:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:49.793 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.963910+0000", 2023-11-01T04:16:49.793 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.812123802000002, 2023-11-01T04:16:49.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.410473274, 2023-11-01T04:16:49.793 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.793 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:49.794 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.794 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.794 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.794 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 186 2023-11-01T04:16:49.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.794 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963910+0000", 2023-11-01T04:16:49.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963910+0000", 2023-11-01T04:16:49.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.797 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.797 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963911+0000", 2023-11-01T04:16:49.797 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.28e-06 2023-11-01T04:16:49.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.797 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963917+0000", 2023-11-01T04:16:49.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1829999999999999e-06 2023-11-01T04:16:49.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963942+0000", 2023-11-01T04:16:49.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4808e-05 2023-11-01T04:16:49.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963957+0000", 2023-11-01T04:16:49.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4698999999999999e-05 2023-11-01T04:16:49.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.967838+0000", 2023-11-01T04:16:49.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003881139 2023-11-01T04:16:49.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.801 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.968035+0000", 2023-11-01T04:16:49.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019704400000000001 2023-11-01T04:16:49.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.802 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:49.802 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.968889+0000", 2023-11-01T04:16:49.802 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00085385099999999998 2023-11-01T04:16:49.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.803 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.803 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:49.803 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.072489+0000", 2023-11-01T04:16:49.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.103600028 2023-11-01T04:16:49.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:49.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374278+0000", 2023-11-01T04:16:49.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.301788789 2023-11-01T04:16:49.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374383+0000", 2023-11-01T04:16:49.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010545299999999999 2023-11-01T04:16:49.805 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.806 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.806 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.806 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:187 3.es0 3:723efae4:::smithi19039307-36:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:49.806 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.964019+0000", 2023-11-01T04:16:49.807 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.812015039000002, 2023-11-01T04:16:49.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41093530900000003, 2023-11-01T04:16:49.807 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.807 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:49.807 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.808 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.808 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.808 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 187 2023-11-01T04:16:49.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.808 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964019+0000", 2023-11-01T04:16:49.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964019+0000", 2023-11-01T04:16:49.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964020+0000", 2023-11-01T04:16:49.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.26e-06 2023-11-01T04:16:49.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964026+0000", 2023-11-01T04:16:49.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3260000000000001e-06 2023-11-01T04:16:49.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964049+0000", 2023-11-01T04:16:49.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2255e-05 2023-11-01T04:16:49.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964067+0000", 2023-11-01T04:16:49.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8643000000000001e-05 2023-11-01T04:16:49.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.969573+0000", 2023-11-01T04:16:49.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055055729999999997 2023-11-01T04:16:49.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.969770+0000", 2023-11-01T04:16:49.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000196787 2023-11-01T04:16:49.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:49.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.971048+0000", 2023-11-01T04:16:49.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012779709999999999 2023-11-01T04:16:49.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:49.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.072491+0000", 2023-11-01T04:16:49.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101443751 2023-11-01T04:16:49.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:49.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374869+0000", 2023-11-01T04:16:49.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.302377644 2023-11-01T04:16:49.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374954+0000", 2023-11-01T04:16:49.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5099000000000006e-05 2023-11-01T04:16:49.819 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.819 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.820 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.820 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:194 3.es0 3:706dddd5:::smithi19039307-14:head [copy-from ver 6] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:49.820 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966030+0000", 2023-11-01T04:16:49.820 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.810003506000001, 2023-11-01T04:16:49.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.72279825099999995, 2023-11-01T04:16:49.821 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.821 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:49.821 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.821 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.822 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.822 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 194 2023-11-01T04:16:49.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.822 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966030+0000", 2023-11-01T04:16:49.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966030+0000", 2023-11-01T04:16:49.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966032+0000", 2023-11-01T04:16:49.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9470000000000002e-06 2023-11-01T04:16:49.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966043+0000", 2023-11-01T04:16:49.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0396e-05 2023-11-01T04:16:49.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966073+0000", 2023-11-01T04:16:49.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0518000000000002e-05 2023-11-01T04:16:49.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966093+0000", 2023-11-01T04:16:49.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9955000000000001e-05 2023-11-01T04:16:49.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.071960+0000", 2023-11-01T04:16:49.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.105867171 2023-11-01T04:16:49.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.072114+0000", 2023-11-01T04:16:49.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000153323 2023-11-01T04:16:49.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:49.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.480839+0000", 2023-11-01T04:16:49.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40872525500000001 2023-11-01T04:16:49.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:49.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.487356+0000", 2023-11-01T04:16:49.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0065175770000000001 2023-11-01T04:16:49.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:49.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.688636+0000", 2023-11-01T04:16:49.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20127930899999999 2023-11-01T04:16:49.832 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.832 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.832 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.688829+0000", 2023-11-01T04:16:49.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0001928 2023-11-01T04:16:49.833 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.833 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.833 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.834 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:195 3.es0 3:706dddd5:::smithi19039307-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:49.834 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966129+0000", 2023-11-01T04:16:49.834 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.809905215999997, 2023-11-01T04:16:49.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51772273199999996, 2023-11-01T04:16:49.834 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.834 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:49.835 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.835 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.835 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.835 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 195 2023-11-01T04:16:49.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.835 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966129+0000", 2023-11-01T04:16:49.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.837 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966129+0000", 2023-11-01T04:16:49.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966130+0000", 2023-11-01T04:16:49.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4529999999999999e-06 2023-11-01T04:16:49.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966168+0000", 2023-11-01T04:16:49.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7792e-05 2023-11-01T04:16:49.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966192+0000", 2023-11-01T04:16:49.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4083999999999998e-05 2023-11-01T04:16:49.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.841 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966210+0000", 2023-11-01T04:16:49.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8196e-05 2023-11-01T04:16:49.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.841 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.841 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.841 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.173008+0000", 2023-11-01T04:16:49.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20679805200000001 2023-11-01T04:16:49.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2023-11-01T04:16:49.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.173168+0000", 2023-11-01T04:16:49.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000159457 2023-11-01T04:16:49.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.843 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.843 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.483564+0000", 2023-11-01T04:16:49.844 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31039652000000001 2023-11-01T04:16:49.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.844 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.483711+0000", 2023-11-01T04:16:49.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014698099999999999 2023-11-01T04:16:49.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.845 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.845 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.483851+0000", 2023-11-01T04:16:49.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000140197 2023-11-01T04:16:49.846 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.846 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.846 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.847 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:202 3.es0 3:7f8a06cd:::smithi19039307-49:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:49.847 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966420+0000", 2023-11-01T04:16:49.847 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.809613908000003, 2023-11-01T04:16:49.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.91933000499999995, 2023-11-01T04:16:49.847 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.847 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:49.848 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.848 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.848 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.848 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 202 2023-11-01T04:16:49.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.849 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966420+0000", 2023-11-01T04:16:49.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.850 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.850 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966420+0000", 2023-11-01T04:16:49.850 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966421+0000", 2023-11-01T04:16:49.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6199999999999999e-06 2023-11-01T04:16:49.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.852 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966431+0000", 2023-11-01T04:16:49.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4660000000000005e-06 2023-11-01T04:16:49.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966456+0000", 2023-11-01T04:16:49.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4748000000000001e-05 2023-11-01T04:16:49.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966473+0000", 2023-11-01T04:16:49.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7323e-05 2023-11-01T04:16:49.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.375871+0000", 2023-11-01T04:16:49.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40939844800000003 2023-11-01T04:16:49.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.376008+0000", 2023-11-01T04:16:49.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000136997 2023-11-01T04:16:49.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:49.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.585026+0000", 2023-11-01T04:16:49.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.209017436 2023-11-01T04:16:49.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.857 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:49.857 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.688063+0000", 2023-11-01T04:16:49.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.103037031 2023-11-01T04:16:49.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:49.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.885528+0000", 2023-11-01T04:16:49.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.19746507999999999 2023-11-01T04:16:49.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.885750+0000", 2023-11-01T04:16:49.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022185600000000001 2023-11-01T04:16:49.859 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.860 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.860 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.860 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:203 3.es0 3:7f8a06cd:::smithi19039307-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:49.860 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966500+0000", 2023-11-01T04:16:49.861 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.809533510000001, 2023-11-01T04:16:49.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.62108323700000001, 2023-11-01T04:16:49.861 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.861 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:49.861 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.861 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.862 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.862 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 203 2023-11-01T04:16:49.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.862 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966500+0000", 2023-11-01T04:16:49.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966500+0000", 2023-11-01T04:16:49.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966501+0000", 2023-11-01T04:16:49.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1570000000000001e-06 2023-11-01T04:16:49.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966530+0000", 2023-11-01T04:16:49.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8337999999999999e-05 2023-11-01T04:16:49.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966549+0000", 2023-11-01T04:16:49.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9624999999999999e-05 2023-11-01T04:16:49.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966584+0000", 2023-11-01T04:16:49.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4442e-05 2023-11-01T04:16:49.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.376334+0000", 2023-11-01T04:16:49.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.409750269 2023-11-01T04:16:49.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.869 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2023-11-01T04:16:49.869 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.376468+0000", 2023-11-01T04:16:49.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000133588 2023-11-01T04:16:49.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.869 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.870 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.586951+0000", 2023-11-01T04:16:49.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21048317499999999 2023-11-01T04:16:49.870 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.870 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.587248+0000", 2023-11-01T04:16:49.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029743400000000001 2023-11-01T04:16:49.871 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.871 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.871 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.587583+0000", 2023-11-01T04:16:49.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000335209 2023-11-01T04:16:49.872 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.872 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.872 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.873 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=22, last_complete=37'15, committed=1, applied=1))", 2023-11-01T04:16:49.873 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.073048+0000", 2023-11-01T04:16:49.873 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.702985601999998, 2023-11-01T04:16:49.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30149571800000002, 2023-11-01T04:16:49.873 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.874 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:49.874 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073048+0000", 2023-11-01T04:16:49.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073048+0000", 2023-11-01T04:16:49.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073049+0000", 2023-11-01T04:16:49.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.96e-07 2023-11-01T04:16:49.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073052+0000", 2023-11-01T04:16:49.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9739999999999998e-06 2023-11-01T04:16:49.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073061+0000", 2023-11-01T04:16:49.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.003e-06 2023-11-01T04:16:49.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073067+0000", 2023-11-01T04:16:49.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6560000000000003e-06 2023-11-01T04:16:49.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374508+0000", 2023-11-01T04:16:49.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30144055400000003 2023-11-01T04:16:49.880 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.880 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.880 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.880 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374544+0000", 2023-11-01T04:16:49.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6134999999999999e-05 2023-11-01T04:16:49.881 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.881 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.881 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.882 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=21, last_complete=37'14, committed=1, applied=1))", 2023-11-01T04:16:49.882 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.073145+0000", 2023-11-01T04:16:49.882 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.702888387999998, 2023-11-01T04:16:49.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30125827300000002, 2023-11-01T04:16:49.882 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.882 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:49.883 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073145+0000", 2023-11-01T04:16:49.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073145+0000", 2023-11-01T04:16:49.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073146+0000", 2023-11-01T04:16:49.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8599999999999999e-07 2023-11-01T04:16:49.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.886 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.886 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073149+0000", 2023-11-01T04:16:49.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8339999999999999e-06 2023-11-01T04:16:49.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.887 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.887 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073160+0000", 2023-11-01T04:16:49.887 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1671999999999999e-05 2023-11-01T04:16:49.887 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.887 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073168+0000", 2023-11-01T04:16:49.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2269999999999996e-06 2023-11-01T04:16:49.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.889 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374161+0000", 2023-11-01T04:16:49.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30099383499999999 2023-11-01T04:16:49.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.889 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.889 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374404+0000", 2023-11-01T04:16:49.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024231900000000001 2023-11-01T04:16:49.890 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.890 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.890 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.891 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 37/31 ECSubWriteReply(tid=22, last_complete=37'15, committed=1, applied=1))", 2023-11-01T04:16:49.891 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:17.073177+0000", 2023-11-01T04:16:49.891 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.702856457000003, 2023-11-01T04:16:49.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30259569400000003, 2023-11-01T04:16:49.891 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.892 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:49.892 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073177+0000", 2023-11-01T04:16:49.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073177+0000", 2023-11-01T04:16:49.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.894 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.894 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.894 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073178+0000", 2023-11-01T04:16:49.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6300000000000001e-07 2023-11-01T04:16:49.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.895 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.895 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073180+0000", 2023-11-01T04:16:49.895 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.65e-06 2023-11-01T04:16:49.895 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.896 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.896 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073188+0000", 2023-11-01T04:16:49.896 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0730000000000005e-06 2023-11-01T04:16:49.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.896 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.896 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.073194+0000", 2023-11-01T04:16:49.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1800000000000001e-06 2023-11-01T04:16:49.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374769+0000", 2023-11-01T04:16:49.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30157417600000003 2023-11-01T04:16:49.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.898 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.375773+0000", 2023-11-01T04:16:49.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001004352 2023-11-01T04:16:49.899 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.899 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.899 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.900 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:271 3.es0 3:706dddd5:::smithi19039307-14:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-01T04:16:49.900 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.017519+0000", 2023-11-01T04:16:49.900 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 28.758514644999998, 2023-11-01T04:16:49.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31206822200000001, 2023-11-01T04:16:49.900 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.900 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:49.901 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.901 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.901 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.901 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 271 2023-11-01T04:16:49.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.902 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.902 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.902 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017519+0000", 2023-11-01T04:16:49.902 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.902 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.903 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.903 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017519+0000", 2023-11-01T04:16:49.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.904 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017520+0000", 2023-11-01T04:16:49.904 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.203e-06 2023-11-01T04:16:49.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017527+0000", 2023-11-01T04:16:49.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7089999999999996e-06 2023-11-01T04:16:49.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.905 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017559+0000", 2023-11-01T04:16:49.906 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1562e-05 2023-11-01T04:16:49.906 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.906 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017568+0000", 2023-11-01T04:16:49.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7189999999999996e-06 2023-11-01T04:16:49.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.907 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.907 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017754+0000", 2023-11-01T04:16:49.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018541 2023-11-01T04:16:49.908 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.908 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.908 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:49.908 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017764+0000", 2023-11-01T04:16:49.908 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0609e-05 2023-11-01T04:16:49.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.023462+0000", 2023-11-01T04:16:49.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0056980370000000004 2023-11-01T04:16:49.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.023706+0000", 2023-11-01T04:16:49.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024344300000000001 2023-11-01T04:16:49.910 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.911 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.329587+0000", 2023-11-01T04:16:49.911 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30588153000000001 2023-11-01T04:16:49.911 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.911 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.912 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.912 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:272 3.es0 3:706dddd5:::smithi19039307-14:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-01T04:16:49.912 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.017583+0000", 2023-11-01T04:16:49.913 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 28.758450989, 2023-11-01T04:16:49.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.412902926, 2023-11-01T04:16:49.913 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.913 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:49.913 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.913 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.914 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.914 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 272 2023-11-01T04:16:49.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.914 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017583+0000", 2023-11-01T04:16:49.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.916 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017583+0000", 2023-11-01T04:16:49.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.916 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017584+0000", 2023-11-01T04:16:49.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6000000000000002e-07 2023-11-01T04:16:49.917 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017588+0000", 2023-11-01T04:16:49.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1330000000000001e-06 2023-11-01T04:16:49.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017598+0000", 2023-11-01T04:16:49.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8430000000000005e-06 2023-11-01T04:16:49.919 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.919 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017605+0000", 2023-11-01T04:16:49.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7030000000000006e-06 2023-11-01T04:16:49.919 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.920 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.920 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017822+0000", 2023-11-01T04:16:49.920 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000217137 2023-11-01T04:16:49.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.921 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.921 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:49.921 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017832+0000", 2023-11-01T04:16:49.921 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2459999999999992e-06 2023-11-01T04:16:49.921 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.921 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.922 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.226201+0000", 2023-11-01T04:16:49.922 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20836893100000001 2023-11-01T04:16:49.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.226369+0000", 2023-11-01T04:16:49.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016842100000000001 2023-11-01T04:16:49.923 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.924 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.430486+0000", 2023-11-01T04:16:49.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20411665200000001 2023-11-01T04:16:49.924 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.924 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.924 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.925 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 41/31 ECSubWriteReply(tid=39, last_complete=41'29, committed=1, applied=1))", 2023-11-01T04:16:49.925 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.853045+0000", 2023-11-01T04:16:49.925 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.922989112, 2023-11-01T04:16:49.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30061494799999999, 2023-11-01T04:16:49.925 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.926 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:49.926 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853045+0000", 2023-11-01T04:16:49.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.927 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853045+0000", 2023-11-01T04:16:49.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853046+0000", 2023-11-01T04:16:49.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8700000000000004e-07 2023-11-01T04:16:49.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853049+0000", 2023-11-01T04:16:49.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.54e-06 2023-11-01T04:16:49.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853058+0000", 2023-11-01T04:16:49.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1540000000000008e-06 2023-11-01T04:16:49.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.931 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.931 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.853064+0000", 2023-11-01T04:16:49.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6459999999999998e-06 2023-11-01T04:16:49.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.931 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:22.153623+0000", 2023-11-01T04:16:49.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30055869000000002 2023-11-01T04:16:49.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:22.153660+0000", 2023-11-01T04:16:49.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7030999999999998e-05 2023-11-01T04:16:49.933 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.933 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.933 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.934 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:499 3.es0 3:70b27670:::smithi19039307-45:head [read 0~753664,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2023-11-01T04:16:49.934 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:44.209465+0000", 2023-11-01T04:16:49.934 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.5665689140000003, 2023-11-01T04:16:49.934 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30689801900000002, 2023-11-01T04:16:49.934 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.935 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:49.935 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.935 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.935 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.935 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 499 2023-11-01T04:16:49.935 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.936 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.936 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209465+0000", 2023-11-01T04:16:49.936 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.936 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209465+0000", 2023-11-01T04:16:49.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209468+0000", 2023-11-01T04:16:49.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7499999999999999e-06 2023-11-01T04:16:49.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209480+0000", 2023-11-01T04:16:49.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2869e-05 2023-11-01T04:16:49.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209514+0000", 2023-11-01T04:16:49.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3436000000000003e-05 2023-11-01T04:16:49.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.940 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.941 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209531+0000", 2023-11-01T04:16:49.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6869000000000001e-05 2023-11-01T04:16:49.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.941 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.941 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209609+0000", 2023-11-01T04:16:49.942 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8323000000000002e-05 2023-11-01T04:16:49.942 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.942 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.942 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2023-11-01T04:16:49.942 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209615+0000", 2023-11-01T04:16:49.942 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.083e-06 2023-11-01T04:16:49.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.943 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.213513+0000", 2023-11-01T04:16:49.943 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0038975379999999999 2023-11-01T04:16:49.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.213655+0000", 2023-11-01T04:16:49.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014228 2023-11-01T04:16:49.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.516363+0000", 2023-11-01T04:16:49.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30270787100000002 2023-11-01T04:16:49.945 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.945 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.946 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.946 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.946 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:500 3.es0 3:70b27670:::smithi19039307-45:head [read 0~753664,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2023-11-01T04:16:49.946 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:44.209565+0000", 2023-11-01T04:16:49.947 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.5664687309999996, 2023-11-01T04:16:49.947 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30789424900000001, 2023-11-01T04:16:49.947 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.947 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:49.947 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.947 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.948 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.948 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 500 2023-11-01T04:16:49.948 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.948 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.948 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.948 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209565+0000", 2023-11-01T04:16:49.949 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.949 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209565+0000", 2023-11-01T04:16:49.950 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.950 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.951 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209566+0000", 2023-11-01T04:16:49.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8000000000000004e-07 2023-11-01T04:16:49.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.951 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.951 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.951 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209571+0000", 2023-11-01T04:16:49.952 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2900000000000002e-06 2023-11-01T04:16:49.952 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.952 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.952 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.952 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209583+0000", 2023-11-01T04:16:49.952 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1823e-05 2023-11-01T04:16:49.953 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.953 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209592+0000", 2023-11-01T04:16:49.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8070000000000006e-06 2023-11-01T04:16:49.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.954 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.954 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209725+0000", 2023-11-01T04:16:49.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013319199999999999 2023-11-01T04:16:49.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:49.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209739+0000", 2023-11-01T04:16:49.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3653e-05 2023-11-01T04:16:49.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.956 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.956 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.214297+0000", 2023-11-01T04:16:49.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045587960000000004 2023-11-01T04:16:49.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.956 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.214428+0000", 2023-11-01T04:16:49.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013051999999999999 2023-11-01T04:16:49.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.517459+0000", 2023-11-01T04:16:49.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30303128800000001 2023-11-01T04:16:49.958 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.958 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.958 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.959 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:501 3.es0 3:70b27670:::smithi19039307-45:head [read 0~753664,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2023-11-01T04:16:49.959 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:44.209637+0000", 2023-11-01T04:16:49.959 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.566396557, 2023-11-01T04:16:49.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30884200699999997, 2023-11-01T04:16:49.960 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.960 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:49.960 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.960 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.960 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.960 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 501 2023-11-01T04:16:49.961 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.961 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.961 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.961 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.961 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209637+0000", 2023-11-01T04:16:49.962 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.962 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.962 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.962 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.962 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209637+0000", 2023-11-01T04:16:49.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209638+0000", 2023-11-01T04:16:49.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4000000000000001e-07 2023-11-01T04:16:49.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209642+0000", 2023-11-01T04:16:49.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9700000000000001e-06 2023-11-01T04:16:49.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.209657+0000", 2023-11-01T04:16:49.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4783999999999999e-05 2023-11-01T04:16:49.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.309802+0000", 2023-11-01T04:16:49.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10014522300000001 2023-11-01T04:16:49.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.310019+0000", 2023-11-01T04:16:49.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000217254 2023-11-01T04:16:49.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.310189+0000", 2023-11-01T04:16:49.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000169481 2023-11-01T04:16:49.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.518479+0000", 2023-11-01T04:16:49.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20829055499999999 2023-11-01T04:16:49.969 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.969 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.969 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.970 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.970 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:507 3.es0 3:7f8a06cd:::smithi19039307-49:head [read 0~1114112,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2023-11-01T04:16:49.970 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:44.309857+0000", 2023-11-01T04:16:49.970 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.4661768649999996, 2023-11-01T04:16:49.970 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31000852000000001, 2023-11-01T04:16:49.971 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.971 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:49.971 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.971 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.971 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.971 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 507 2023-11-01T04:16:49.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.972 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.309857+0000", 2023-11-01T04:16:49.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.973 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.973 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.309857+0000", 2023-11-01T04:16:49.973 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.974 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.974 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.974 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.309859+0000", 2023-11-01T04:16:49.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9599999999999999e-06 2023-11-01T04:16:49.974 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.975 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.975 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.309872+0000", 2023-11-01T04:16:49.975 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3433e-05 2023-11-01T04:16:49.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.975 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.309896+0000", 2023-11-01T04:16:49.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3238999999999999e-05 2023-11-01T04:16:49.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.309908+0000", 2023-11-01T04:16:49.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2193e-05 2023-11-01T04:16:49.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.410829+0000", 2023-11-01T04:16:49.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100921653 2023-11-01T04:16:49.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.410992+0000", 2023-11-01T04:16:49.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016277899999999999 2023-11-01T04:16:49.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.619865+0000", 2023-11-01T04:16:49.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.208873263 2023-11-01T04:16:49.980 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.980 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.980 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.981 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:508 3.es0 3:7f8a06cd:::smithi19039307-49:head [read 0~1114112,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2023-11-01T04:16:49.981 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:44.309990+0000", 2023-11-01T04:16:49.981 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.4660434389999999, 2023-11-01T04:16:49.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31257325499999999, 2023-11-01T04:16:49.981 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.981 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:49.982 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.982 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.982 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.982 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 508 2023-11-01T04:16:49.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.982 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.309990+0000", 2023-11-01T04:16:49.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.984 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.984 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.309990+0000", 2023-11-01T04:16:49.984 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.985 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.985 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.309991+0000", 2023-11-01T04:16:49.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7000000000000003e-07 2023-11-01T04:16:49.985 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.985 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.985 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.310014+0000", 2023-11-01T04:16:49.986 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3263000000000001e-05 2023-11-01T04:16:49.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.986 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.986 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.310030+0000", 2023-11-01T04:16:49.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5835999999999999e-05 2023-11-01T04:16:49.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.310040+0000", 2023-11-01T04:16:49.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.363e-06 2023-11-01T04:16:49.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:49.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.411497+0000", 2023-11-01T04:16:49.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10145696999999999 2023-11-01T04:16:49.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.989 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:49.989 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.411622+0000", 2023-11-01T04:16:49.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012524300000000001 2023-11-01T04:16:49.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.990 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:49.990 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.622564+0000", 2023-11-01T04:16:49.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21094171 2023-11-01T04:16:49.990 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.990 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:49.991 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:49.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.991 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:509 3.es0 3:7f8a06cd:::smithi19039307-49:head [read 0~1114112,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2023-11-01T04:16:49.991 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:44.310053+0000", 2023-11-01T04:16:49.992 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.4659804669999996, 2023-11-01T04:16:49.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41447947000000002, 2023-11-01T04:16:49.992 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:49.992 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:16:49.993 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:16:49.993 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:16:49.993 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:49.993 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 509 2023-11-01T04:16:49.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.994 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:49.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:49.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.310053+0000", 2023-11-01T04:16:49.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:49.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.310053+0000", 2023-11-01T04:16:49.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:49.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:49.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.310054+0000", 2023-11-01T04:16:49.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7999999999999993e-07 2023-11-01T04:16:49.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:49.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.310058+0000", 2023-11-01T04:16:49.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.019e-06 2023-11-01T04:16:49.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:49.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.310067+0000", 2023-11-01T04:16:49.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1730000000000001e-06 2023-11-01T04:16:49.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:49.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.410163+0000", 2023-11-01T04:16:49.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100095695 2023-11-01T04:16:49.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:49.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:49.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.620051+0000", 2023-11-01T04:16:50.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20988817600000001 2023-11-01T04:16:50.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.620336+0000", 2023-11-01T04:16:50.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028468100000000001 2023-11-01T04:16:50.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.724533+0000", 2023-11-01T04:16:50.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10419674600000001 2023-11-01T04:16:50.002 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.002 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.002 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.003 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.es0 67/31 ECSubWriteReply(tid=76, last_complete=67'56, committed=1, applied=1))", 2023-11-01T04:16:50.003 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:47.601946+0000", 2023-11-01T04:16:50.003 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 2.1740881779999999, 2023-11-01T04:16:50.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30267189, 2023-11-01T04:16:50.003 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.004 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:50.004 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.601946+0000", 2023-11-01T04:16:50.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.601946+0000", 2023-11-01T04:16:50.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.601955+0000", 2023-11-01T04:16:50.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3889999999999994e-06 2023-11-01T04:16:50.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.601967+0000", 2023-11-01T04:16:50.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.166e-05 2023-11-01T04:16:50.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.601990+0000", 2023-11-01T04:16:50.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3609e-05 2023-11-01T04:16:50.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.702082+0000", 2023-11-01T04:16:50.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10009156900000001 2023-11-01T04:16:50.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.904580+0000", 2023-11-01T04:16:50.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20249788099999999 2023-11-01T04:16:50.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:47.904617+0000", 2023-11-01T04:16:50.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7781999999999998e-05 2023-11-01T04:16:50.013 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.013 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.013 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.014 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.014 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.015 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:50.016 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-01T04:16:50.018 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:50.016+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:50.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-01T04:16:50.021 INFO:tasks.rados.rados.0.smithi190.stdout:230: finishing write tid 2 to smithi19039307-14 2023-11-01T04:16:50.021 INFO:tasks.rados.rados.0.smithi190.stdout:230: finishing write tid 3 to smithi19039307-14 2023-11-01T04:16:50.021 INFO:tasks.rados.rados.0.smithi190.stdout:230: finishing write tid 4 to smithi19039307-14 2023-11-01T04:16:50.021 INFO:tasks.rados.rados.0.smithi190.stdout:228: done (4 left) 2023-11-01T04:16:50.022 INFO:tasks.rados.rados.0.smithi190.stdout:233: done (3 left) 2023-11-01T04:16:50.022 INFO:tasks.rados.rados.0.smithi190.stdout:234: rollback oid 49 current snap is 21 2023-11-01T04:16:50.022 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 49 to 12 2023-11-01T04:16:50.022 INFO:tasks.rados.rados.0.smithi190.stdout:235: delete oid 39 current snap is 21 2023-11-01T04:16:50.033 INFO:tasks.rados.rados.0.smithi190.stdout:232: finishing write tid 1 to smithi19039307-24 2023-11-01T04:16:50.034 INFO:tasks.rados.rados.0.smithi190.stdout:232: finishing write tid 2 to smithi19039307-24 2023-11-01T04:16:50.034 INFO:tasks.rados.rados.0.smithi190.stdout:232: finishing write tid 3 to smithi19039307-24 2023-11-01T04:16:50.118 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:50.116+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:50.126 INFO:tasks.rados.rados.0.smithi190.stdout:235: done (4 left) 2023-11-01T04:16:50.126 INFO:tasks.rados.rados.0.smithi190.stdout:236: copy_from oid 36 from oid 32 current snap is 21 2023-11-01T04:16:50.126 INFO:tasks.rados.rados.0.smithi190.stdout:237: rollback oid 27 current snap is 21 2023-11-01T04:16:50.127 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 27 to 12 2023-11-01T04:16:50.127 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 13 current snap is 21 2023-11-01T04:16:50.127 INFO:tasks.rados.rados.0.smithi190.stdout:238: seq_num 98 ranges {1605632=491520,2097152=589824,2686976=557056,3244032=16384} 2023-11-01T04:16:50.139 INFO:tasks.rados.rados.0.smithi190.stdout:238: writing smithi19039307-13 from 1605632 to 2097152 tid 1 2023-11-01T04:16:50.143 INFO:tasks.rados.rados.0.smithi190.stdout:238: writing smithi19039307-13 from 2097152 to 2686976 tid 2 2023-11-01T04:16:50.147 INFO:tasks.rados.rados.0.smithi190.stdout:238: writing smithi19039307-13 from 2686976 to 3244032 tid 3 2023-11-01T04:16:50.147 INFO:tasks.rados.rados.0.smithi190.stdout:238: writing smithi19039307-13 from 3244032 to 3260416 tid 4 2023-11-01T04:16:50.148 INFO:tasks.rados.rados.0.smithi190.stdout:239: rollback oid 25 current snap is 21 2023-11-01T04:16:50.148 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 25 to 20 2023-11-01T04:16:50.148 INFO:tasks.rados.rados.0.smithi190.stdout:240: snap_remove snap 19 2023-11-01T04:16:50.162 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:50.162 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:50.163 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2023-11-01T04:16:50.163 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:50.176 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:50.176 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-11-01T04:16:50.177 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-01T04:16:50.196 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:50.197 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-01T04:16:50.216 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:50.217 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-01T04:16:50.220 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:50.218+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:50.235 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:50.235 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-01T04:16:50.320 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:50.317+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:50.344 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:50.344 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:16:50.344 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2023-11-01T04:16:50.344 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2023-11-01T04:16:50.344 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:50.348 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:50.349 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-01T04:16:50.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-01T04:16:50.377 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:50.378 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-01T04:16:50.405 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:50.405 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-01T04:16:50.421 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:50.420+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:50.431 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:50.432 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-01T04:16:50.457 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:50.458 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-01T04:16:50.482 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:50.482 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-01T04:16:50.483 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:16:50.484 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2023-11-01T04:16:50.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2023-11-01T04:16:50.484 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2023-11-01T04:16:50.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.484 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:36 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'22, mlcod=25'16)", 2023-11-01T04:16:50.485 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:00.688757+0000", 2023-11-01T04:16:50.485 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 49.794910334000001, 2023-11-01T04:16:50.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20189465300000001, 2023-11-01T04:16:50.485 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.485 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.485 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688757+0000", 2023-11-01T04:16:50.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.487 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688757+0000", 2023-11-01T04:16:50.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.488 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.488 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688758+0000", 2023-11-01T04:16:50.488 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5300000000000004e-07 2023-11-01T04:16:50.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.488 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688763+0000", 2023-11-01T04:16:50.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7389999999999996e-06 2023-11-01T04:16:50.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688791+0000", 2023-11-01T04:16:50.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8136999999999998e-05 2023-11-01T04:16:50.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688799+0000", 2023-11-01T04:16:50.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3390000000000004e-06 2023-11-01T04:16:50.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.888843+0000", 2023-11-01T04:16:50.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20004432999999999 2023-11-01T04:16:50.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.888878+0000", 2023-11-01T04:16:50.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4814000000000002e-05 2023-11-01T04:16:50.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.890502+0000", 2023-11-01T04:16:50.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001623813 2023-11-01T04:16:50.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.890652+0000", 2023-11-01T04:16:50.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014982799999999999 2023-11-01T04:16:50.494 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.495 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.495 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.495 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:37 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'23, mlcod=25'16)", 2023-11-01T04:16:50.495 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:00.688817+0000", 2023-11-01T04:16:50.496 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 49.794850533999998, 2023-11-01T04:16:50.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.202682319, 2023-11-01T04:16:50.496 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.496 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.496 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688817+0000", 2023-11-01T04:16:50.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688817+0000", 2023-11-01T04:16:50.498 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.499 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688817+0000", 2023-11-01T04:16:50.499 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6700000000000003e-07 2023-11-01T04:16:50.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.499 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.499 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688823+0000", 2023-11-01T04:16:50.500 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1589999999999998e-06 2023-11-01T04:16:50.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.500 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688835+0000", 2023-11-01T04:16:50.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1472e-05 2023-11-01T04:16:50.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.788922+0000", 2023-11-01T04:16:50.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100087538 2023-11-01T04:16:50.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.889458+0000", 2023-11-01T04:16:50.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100535469 2023-11-01T04:16:50.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.889515+0000", 2023-11-01T04:16:50.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6990999999999997e-05 2023-11-01T04:16:50.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.891416+0000", 2023-11-01T04:16:50.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019012409999999999 2023-11-01T04:16:50.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.505 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.891499+0000", 2023-11-01T04:16:50.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2881999999999997e-05 2023-11-01T04:16:50.505 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.506 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.506 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.506 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:53 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'31, mlcod=26'29)", 2023-11-01T04:16:50.507 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:01.627435+0000", 2023-11-01T04:16:50.507 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 48.856232235999997, 2023-11-01T04:16:50.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20423751000000001, 2023-11-01T04:16:50.507 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.507 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.507 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627435+0000", 2023-11-01T04:16:50.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627435+0000", 2023-11-01T04:16:50.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627437+0000", 2023-11-01T04:16:50.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6670000000000001e-06 2023-11-01T04:16:50.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627449+0000", 2023-11-01T04:16:50.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2571999999999999e-05 2023-11-01T04:16:50.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627477+0000", 2023-11-01T04:16:50.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.741e-05 2023-11-01T04:16:50.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.512 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627491+0000", 2023-11-01T04:16:50.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3953e-05 2023-11-01T04:16:50.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627659+0000", 2023-11-01T04:16:50.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016794299999999999 2023-11-01T04:16:50.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627699+0000", 2023-11-01T04:16:50.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0036000000000001e-05 2023-11-01T04:16:50.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831530+0000", 2023-11-01T04:16:50.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20383152299999999 2023-11-01T04:16:50.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831672+0000", 2023-11-01T04:16:50.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014240600000000001 2023-11-01T04:16:50.516 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.517 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.517 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.517 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:54 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'32, mlcod=26'29)", 2023-11-01T04:16:50.518 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:01.628360+0000", 2023-11-01T04:16:50.518 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 48.855307457000002, 2023-11-01T04:16:50.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20340096899999999, 2023-11-01T04:16:50.518 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.518 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.518 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.519 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628360+0000", 2023-11-01T04:16:50.519 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628360+0000", 2023-11-01T04:16:50.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628362+0000", 2023-11-01T04:16:50.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.787e-06 2023-11-01T04:16:50.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628374+0000", 2023-11-01T04:16:50.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2499e-05 2023-11-01T04:16:50.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628399+0000", 2023-11-01T04:16:50.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5037000000000001e-05 2023-11-01T04:16:50.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628412+0000", 2023-11-01T04:16:50.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2982e-05 2023-11-01T04:16:50.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628555+0000", 2023-11-01T04:16:50.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000142783 2023-11-01T04:16:50.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.525 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.525 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628587+0000", 2023-11-01T04:16:50.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2026999999999997e-05 2023-11-01T04:16:50.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.526 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831677+0000", 2023-11-01T04:16:50.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20309039200000001 2023-11-01T04:16:50.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831761+0000", 2023-11-01T04:16:50.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3461999999999995e-05 2023-11-01T04:16:50.527 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.528 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.528 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.528 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:55 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'33, mlcod=26'29)", 2023-11-01T04:16:50.528 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:01.629257+0000", 2023-11-01T04:16:50.529 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 48.854410465000001, 2023-11-01T04:16:50.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20259303100000001, 2023-11-01T04:16:50.529 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.529 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.529 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629257+0000", 2023-11-01T04:16:50.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629257+0000", 2023-11-01T04:16:50.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629259+0000", 2023-11-01T04:16:50.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8530000000000001e-06 2023-11-01T04:16:50.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629358+0000", 2023-11-01T04:16:50.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9019999999999997e-05 2023-11-01T04:16:50.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629386+0000", 2023-11-01T04:16:50.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8453999999999999e-05 2023-11-01T04:16:50.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629401+0000", 2023-11-01T04:16:50.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4465e-05 2023-11-01T04:16:50.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.535 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.829838+0000", 2023-11-01T04:16:50.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20043709100000001 2023-11-01T04:16:50.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.829873+0000", 2023-11-01T04:16:50.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5253000000000003e-05 2023-11-01T04:16:50.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.537 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831769+0000", 2023-11-01T04:16:50.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018957100000000001 2023-11-01T04:16:50.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831850+0000", 2023-11-01T04:16:50.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1185000000000003e-05 2023-11-01T04:16:50.538 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.538 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.539 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.539 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:104 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'66, mlcod=28'64)", 2023-11-01T04:16:50.539 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.140797+0000", 2023-11-01T04:16:50.540 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 45.342870173000001, 2023-11-01T04:16:50.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20170181600000001, 2023-11-01T04:16:50.540 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.540 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.540 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.140797+0000", 2023-11-01T04:16:50.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.140797+0000", 2023-11-01T04:16:50.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.140799+0000", 2023-11-01T04:16:50.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.176e-06 2023-11-01T04:16:50.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.140813+0000", 2023-11-01T04:16:50.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3791999999999999e-05 2023-11-01T04:16:50.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.140846+0000", 2023-11-01T04:16:50.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2824000000000003e-05 2023-11-01T04:16:50.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.240986+0000", 2023-11-01T04:16:50.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100140625 2023-11-01T04:16:50.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.546 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241142+0000", 2023-11-01T04:16:50.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015555899999999999 2023-11-01T04:16:50.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241199+0000", 2023-11-01T04:16:50.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7321999999999998e-05 2023-11-01T04:16:50.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.342429+0000", 2023-11-01T04:16:50.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101229401 2023-11-01T04:16:50.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.342499+0000", 2023-11-01T04:16:50.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0117000000000006e-05 2023-11-01T04:16:50.549 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.549 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.549 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.550 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:105 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'67, mlcod=28'64)", 2023-11-01T04:16:50.550 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.241023+0000", 2023-11-01T04:16:50.550 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 45.242643737000002, 2023-11-01T04:16:50.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10730198000000001, 2023-11-01T04:16:50.551 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.551 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.551 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241023+0000", 2023-11-01T04:16:50.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.553 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241023+0000", 2023-11-01T04:16:50.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.553 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241024+0000", 2023-11-01T04:16:50.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9999999999999995e-07 2023-11-01T04:16:50.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241045+0000", 2023-11-01T04:16:50.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0684999999999999e-05 2023-11-01T04:16:50.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.555 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.555 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241062+0000", 2023-11-01T04:16:50.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6956000000000002e-05 2023-11-01T04:16:50.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241072+0000", 2023-11-01T04:16:50.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0329000000000001e-05 2023-11-01T04:16:50.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.341337+0000", 2023-11-01T04:16:50.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100264753 2023-11-01T04:16:50.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.341371+0000", 2023-11-01T04:16:50.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4241e-05 2023-11-01T04:16:50.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.348245+0000", 2023-11-01T04:16:50.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068737700000000004 2023-11-01T04:16:50.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.348325+0000", 2023-11-01T04:16:50.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0246000000000004e-05 2023-11-01T04:16:50.560 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.560 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.561 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.561 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:106 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'68, mlcod=28'64)", 2023-11-01T04:16:50.561 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.241117+0000", 2023-11-01T04:16:50.561 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 45.242549754999999, 2023-11-01T04:16:50.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10725122099999999, 2023-11-01T04:16:50.562 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.562 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.562 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241117+0000", 2023-11-01T04:16:50.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241117+0000", 2023-11-01T04:16:50.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241118+0000", 2023-11-01T04:16:50.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8400000000000004e-07 2023-11-01T04:16:50.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241174+0000", 2023-11-01T04:16:50.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6097999999999997e-05 2023-11-01T04:16:50.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241190+0000", 2023-11-01T04:16:50.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5545000000000001e-05 2023-11-01T04:16:50.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.567 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241201+0000", 2023-11-01T04:16:50.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1148999999999999e-05 2023-11-01T04:16:50.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.341896+0000", 2023-11-01T04:16:50.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100695009 2023-11-01T04:16:50.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.341927+0000", 2023-11-01T04:16:50.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1597000000000003e-05 2023-11-01T04:16:50.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.348328+0000", 2023-11-01T04:16:50.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006401042 2023-11-01T04:16:50.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.348369+0000", 2023-11-01T04:16:50.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0197000000000002e-05 2023-11-01T04:16:50.571 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.572 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.572 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.572 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:107 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'69, mlcod=28'64)", 2023-11-01T04:16:50.572 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.241214+0000", 2023-11-01T04:16:50.573 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 45.24245277, 2023-11-01T04:16:50.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.109683828, 2023-11-01T04:16:50.573 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.573 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.573 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241214+0000", 2023-11-01T04:16:50.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241214+0000", 2023-11-01T04:16:50.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241215+0000", 2023-11-01T04:16:50.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7999999999999995e-07 2023-11-01T04:16:50.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241224+0000", 2023-11-01T04:16:50.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1630000000000002e-06 2023-11-01T04:16:50.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241233+0000", 2023-11-01T04:16:50.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3520000000000007e-06 2023-11-01T04:16:50.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241239+0000", 2023-11-01T04:16:50.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1890000000000004e-06 2023-11-01T04:16:50.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.342619+0000", 2023-11-01T04:16:50.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101380392 2023-11-01T04:16:50.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.342679+0000", 2023-11-01T04:16:50.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0387999999999998e-05 2023-11-01T04:16:50.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350812+0000", 2023-11-01T04:16:50.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0081325249999999998 2023-11-01T04:16:50.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350898+0000", 2023-11-01T04:16:50.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6238999999999999e-05 2023-11-01T04:16:50.582 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.583 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.583 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.583 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:108 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'70, mlcod=28'64)", 2023-11-01T04:16:50.583 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.242130+0000", 2023-11-01T04:16:50.584 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 45.241537071000003, 2023-11-01T04:16:50.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108814548, 2023-11-01T04:16:50.584 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.584 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.584 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242130+0000", 2023-11-01T04:16:50.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242130+0000", 2023-11-01T04:16:50.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242132+0000", 2023-11-01T04:16:50.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7799999999999999e-06 2023-11-01T04:16:50.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242144+0000", 2023-11-01T04:16:50.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2201999999999999e-05 2023-11-01T04:16:50.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242174+0000", 2023-11-01T04:16:50.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9484000000000001e-05 2023-11-01T04:16:50.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242192+0000", 2023-11-01T04:16:50.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8862e-05 2023-11-01T04:16:50.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.590 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.343513+0000", 2023-11-01T04:16:50.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101320113 2023-11-01T04:16:50.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.343569+0000", 2023-11-01T04:16:50.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6158e-05 2023-11-01T04:16:50.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350902+0000", 2023-11-01T04:16:50.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0073331500000000001 2023-11-01T04:16:50.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350945+0000", 2023-11-01T04:16:50.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2799000000000001e-05 2023-11-01T04:16:50.593 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.593 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.593 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.594 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:109 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'71, mlcod=28'64)", 2023-11-01T04:16:50.594 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.243545+0000", 2023-11-01T04:16:50.594 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 45.240122479999997, 2023-11-01T04:16:50.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.107439464, 2023-11-01T04:16:50.595 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.595 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.595 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243545+0000", 2023-11-01T04:16:50.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243545+0000", 2023-11-01T04:16:50.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243547+0000", 2023-11-01T04:16:50.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8530000000000001e-06 2023-11-01T04:16:50.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243603+0000", 2023-11-01T04:16:50.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6498e-05 2023-11-01T04:16:50.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243632+0000", 2023-11-01T04:16:50.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9135000000000001e-05 2023-11-01T04:16:50.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.600 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243646+0000", 2023-11-01T04:16:50.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3845e-05 2023-11-01T04:16:50.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.344360+0000", 2023-11-01T04:16:50.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10071392699999999 2023-11-01T04:16:50.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.602 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.344410+0000", 2023-11-01T04:16:50.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0078999999999999e-05 2023-11-01T04:16:50.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.603 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.603 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350947+0000", 2023-11-01T04:16:50.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0065365129999999999 2023-11-01T04:16:50.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.603 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350984+0000", 2023-11-01T04:16:50.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7614000000000003e-05 2023-11-01T04:16:50.604 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.604 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.604 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.605 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:110 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'72, mlcod=28'64)", 2023-11-01T04:16:50.605 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.244933+0000", 2023-11-01T04:16:50.605 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 45.238734512999997, 2023-11-01T04:16:50.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.107682845, 2023-11-01T04:16:50.606 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.606 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.606 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244933+0000", 2023-11-01T04:16:50.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244933+0000", 2023-11-01T04:16:50.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.608 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.608 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244934+0000", 2023-11-01T04:16:50.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.776e-06 2023-11-01T04:16:50.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244947+0000", 2023-11-01T04:16:50.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2046e-05 2023-11-01T04:16:50.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244976+0000", 2023-11-01T04:16:50.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9141000000000001e-05 2023-11-01T04:16:50.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244989+0000", 2023-11-01T04:16:50.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3675e-05 2023-11-01T04:16:50.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.345397+0000", 2023-11-01T04:16:50.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100407617 2023-11-01T04:16:50.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.345451+0000", 2023-11-01T04:16:50.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4475999999999998e-05 2023-11-01T04:16:50.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.352527+0000", 2023-11-01T04:16:50.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070755089999999998 2023-11-01T04:16:50.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.352616+0000", 2023-11-01T04:16:50.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8604999999999996e-05 2023-11-01T04:16:50.615 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.615 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.615 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.616 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:111 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'73, mlcod=28'64)", 2023-11-01T04:16:50.616 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.246238+0000", 2023-11-01T04:16:50.616 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 45.237428962000003, 2023-11-01T04:16:50.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10642051399999999, 2023-11-01T04:16:50.617 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.617 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.617 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.246238+0000", 2023-11-01T04:16:50.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.246238+0000", 2023-11-01T04:16:50.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.246240+0000", 2023-11-01T04:16:50.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.776e-06 2023-11-01T04:16:50.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.246252+0000", 2023-11-01T04:16:50.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1671999999999999e-05 2023-11-01T04:16:50.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.246280+0000", 2023-11-01T04:16:50.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8551000000000002e-05 2023-11-01T04:16:50.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.246294+0000", 2023-11-01T04:16:50.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3583e-05 2023-11-01T04:16:50.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.346176+0000", 2023-11-01T04:16:50.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.09988242 2023-11-01T04:16:50.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.346213+0000", 2023-11-01T04:16:50.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7209999999999998e-05 2023-11-01T04:16:50.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.352619+0000", 2023-11-01T04:16:50.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064055220000000003 2023-11-01T04:16:50.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.352659+0000", 2023-11-01T04:16:50.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9780000000000002e-05 2023-11-01T04:16:50.626 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.626 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.627 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.627 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:118 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'80, mlcod=28'78)", 2023-11-01T04:16:50.627 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.860484+0000", 2023-11-01T04:16:50.628 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 44.623182859000003, 2023-11-01T04:16:50.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20313931199999999, 2023-11-01T04:16:50.628 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.628 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.628 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860484+0000", 2023-11-01T04:16:50.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860484+0000", 2023-11-01T04:16:50.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860487+0000", 2023-11-01T04:16:50.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5239999999999999e-06 2023-11-01T04:16:50.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860560+0000", 2023-11-01T04:16:50.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3109999999999996e-05 2023-11-01T04:16:50.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860593+0000", 2023-11-01T04:16:50.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3077000000000002e-05 2023-11-01T04:16:50.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860611+0000", 2023-11-01T04:16:50.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7881999999999999e-05 2023-11-01T04:16:50.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.960884+0000", 2023-11-01T04:16:50.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100273563 2023-11-01T04:16:50.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.960949+0000", 2023-11-01T04:16:50.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4250000000000003e-05 2023-11-01T04:16:50.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063550+0000", 2023-11-01T04:16:50.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.102601553 2023-11-01T04:16:50.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063624+0000", 2023-11-01T04:16:50.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3353000000000001e-05 2023-11-01T04:16:50.637 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.637 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.638 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.638 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:119 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'81, mlcod=28'78)", 2023-11-01T04:16:50.638 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.861924+0000", 2023-11-01T04:16:50.639 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 44.621743682999998, 2023-11-01T04:16:50.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.201754672, 2023-11-01T04:16:50.639 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.639 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.639 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.861924+0000", 2023-11-01T04:16:50.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.861924+0000", 2023-11-01T04:16:50.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.861925+0000", 2023-11-01T04:16:50.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.663e-06 2023-11-01T04:16:50.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.862002+0000", 2023-11-01T04:16:50.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7217000000000003e-05 2023-11-01T04:16:50.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.862024+0000", 2023-11-01T04:16:50.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1957999999999999e-05 2023-11-01T04:16:50.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.862035+0000", 2023-11-01T04:16:50.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0482000000000001e-05 2023-11-01T04:16:50.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.961997+0000", 2023-11-01T04:16:50.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.099962336999999998 2023-11-01T04:16:50.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.962032+0000", 2023-11-01T04:16:50.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4804000000000001e-05 2023-11-01T04:16:50.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063634+0000", 2023-11-01T04:16:50.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101602151 2023-11-01T04:16:50.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063678+0000", 2023-11-01T04:16:50.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4060000000000002e-05 2023-11-01T04:16:50.648 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.648 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.648 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.649 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:120 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'82, mlcod=28'78)", 2023-11-01T04:16:50.649 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.863328+0000", 2023-11-01T04:16:50.649 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 44.620339211000001, 2023-11-01T04:16:50.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20039775900000001, 2023-11-01T04:16:50.650 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.650 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.650 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863328+0000", 2023-11-01T04:16:50.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863328+0000", 2023-11-01T04:16:50.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863330+0000", 2023-11-01T04:16:50.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7770000000000001e-06 2023-11-01T04:16:50.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863442+0000", 2023-11-01T04:16:50.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112596 2023-11-01T04:16:50.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863472+0000", 2023-11-01T04:16:50.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9490999999999999e-05 2023-11-01T04:16:50.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863486+0000", 2023-11-01T04:16:50.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3689000000000001e-05 2023-11-01T04:16:50.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.962703+0000", 2023-11-01T04:16:50.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.099217692999999996 2023-11-01T04:16:50.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.962756+0000", 2023-11-01T04:16:50.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2454999999999998e-05 2023-11-01T04:16:50.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063680+0000", 2023-11-01T04:16:50.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100924298 2023-11-01T04:16:50.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063726+0000", 2023-11-01T04:16:50.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5760000000000002e-05 2023-11-01T04:16:50.659 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.659 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.659 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.660 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:121 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'83, mlcod=28'78)", 2023-11-01T04:16:50.660 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.864836+0000", 2023-11-01T04:16:50.660 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 44.618831385, 2023-11-01T04:16:50.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20010170799999999, 2023-11-01T04:16:50.661 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.661 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.661 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864836+0000", 2023-11-01T04:16:50.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864836+0000", 2023-11-01T04:16:50.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864838+0000", 2023-11-01T04:16:50.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7999999999999999e-06 2023-11-01T04:16:50.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864956+0000", 2023-11-01T04:16:50.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000118293 2023-11-01T04:16:50.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864985+0000", 2023-11-01T04:16:50.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9187999999999999e-05 2023-11-01T04:16:50.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864999+0000", 2023-11-01T04:16:50.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3718e-05 2023-11-01T04:16:50.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.062876+0000", 2023-11-01T04:16:50.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.197877154 2023-11-01T04:16:50.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.062909+0000", 2023-11-01T04:16:50.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3059999999999999e-05 2023-11-01T04:16:50.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.064827+0000", 2023-11-01T04:16:50.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019179209999999999 2023-11-01T04:16:50.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.064938+0000", 2023-11-01T04:16:50.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110574 2023-11-01T04:16:50.670 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.670 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.670 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.671 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:133 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'91, mlcod=28'85)", 2023-11-01T04:16:50.671 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:06.724959+0000", 2023-11-01T04:16:50.671 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 43.758707886000003, 2023-11-01T04:16:50.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108020745, 2023-11-01T04:16:50.672 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.672 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.672 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.724959+0000", 2023-11-01T04:16:50.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.724959+0000", 2023-11-01T04:16:50.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.724960+0000", 2023-11-01T04:16:50.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3000000000000001e-07 2023-11-01T04:16:50.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.724965+0000", 2023-11-01T04:16:50.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.463e-06 2023-11-01T04:16:50.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.724978+0000", 2023-11-01T04:16:50.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2748999999999999e-05 2023-11-01T04:16:50.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.825118+0000", 2023-11-01T04:16:50.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100140121 2023-11-01T04:16:50.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.827391+0000", 2023-11-01T04:16:50.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022731800000000001 2023-11-01T04:16:50.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.827428+0000", 2023-11-01T04:16:50.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.65e-05 2023-11-01T04:16:50.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.832911+0000", 2023-11-01T04:16:50.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0054829340000000001 2023-11-01T04:16:50.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.832980+0000", 2023-11-01T04:16:50.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9368000000000004e-05 2023-11-01T04:16:50.681 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.681 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.681 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.682 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:138 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'96, mlcod=28'85)", 2023-11-01T04:16:50.682 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:06.925635+0000", 2023-11-01T04:16:50.682 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 43.558031814000003, 2023-11-01T04:16:50.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20470132099999999, 2023-11-01T04:16:50.682 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.683 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:50.683 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.925635+0000", 2023-11-01T04:16:50.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.925635+0000", 2023-11-01T04:16:50.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.925636+0000", 2023-11-01T04:16:50.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1999999999999999e-07 2023-11-01T04:16:50.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.686 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.686 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.925647+0000", 2023-11-01T04:16:50.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1348999999999999e-05 2023-11-01T04:16:50.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.925660+0000", 2023-11-01T04:16:50.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2782000000000001e-05 2023-11-01T04:16:50.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.025800+0000", 2023-11-01T04:16:50.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100139379 2023-11-01T04:16:50.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.025960+0000", 2023-11-01T04:16:50.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000160063 2023-11-01T04:16:50.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:16:50.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.026036+0000", 2023-11-01T04:16:50.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6637000000000005e-05 2023-11-01T04:16:50.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:16:50.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.130231+0000", 2023-11-01T04:16:50.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.104194433 2023-11-01T04:16:50.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.691 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.130337+0000", 2023-11-01T04:16:50.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010605800000000001 2023-11-01T04:16:50.692 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.692 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.692 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.692 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.4s1 54/31 ECSubWrite(tid=62, reqid=osd.10.0:62, at_version=54'25, trim_to=0'0, roll_forward_to=48'23))", 2023-11-01T04:16:50.693 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:34.226450+0000", 2023-11-01T04:16:50.693 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.257217631, 2023-11-01T04:16:50.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20242526499999999, 2023-11-01T04:16:50.693 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:16:50.693 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:16:50.693 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:16:50.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:16:50.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.226450+0000", 2023-11-01T04:16:50.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:16:50.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.226450+0000", 2023-11-01T04:16:50.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:16:50.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:16:50.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.226451+0000", 2023-11-01T04:16:50.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6700000000000002e-07 2023-11-01T04:16:50.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:16:50.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.226501+0000", 2023-11-01T04:16:50.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0608000000000001e-05 2023-11-01T04:16:50.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:16:50.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.226539+0000", 2023-11-01T04:16:50.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7914999999999999e-05 2023-11-01T04:16:50.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:16:50.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.326654+0000", 2023-11-01T04:16:50.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10011447699999999 2023-11-01T04:16:50.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:16:50.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.426948+0000", 2023-11-01T04:16:50.699 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10029434299999999 2023-11-01T04:16:50.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.700 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:16:50.700 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.426967+0000", 2023-11-01T04:16:50.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9516000000000001e-05 2023-11-01T04:16:50.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:16:50.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.428805+0000", 2023-11-01T04:16:50.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001837991 2023-11-01T04:16:50.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:16:50.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:16:50.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:16:50.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.428875+0000", 2023-11-01T04:16:50.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9448000000000003e-05 2023-11-01T04:16:50.702 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.702 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.703 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.703 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:16:50.703 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:16:50.703 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:16:50.704 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:50.705 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-01T04:16:50.705 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:50.706 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-01T04:16:50.706 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.12 2023-11-01T04:16:50.706 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-01T04:16:50.707 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:50.707 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-01T04:16:50.708 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-01T04:16:50.709 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-01T04:16:50.710 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:50.703+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:50.710 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-01T04:16:50.711 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-01T04:16:50.765 INFO:tasks.rados.rados.0.smithi190.stdout:240: done (8 left) 2023-11-01T04:16:50.766 INFO:tasks.rados.rados.0.smithi190.stdout:241: snap_remove snap 17 2023-11-01T04:16:50.803 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-11-01T04:16:50.805 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:50.801+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:50.878 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:50.878 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:50.894 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-01T04:16:50.905 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:50.902+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:51.007 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:51.002+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:51.010 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:51.011 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:51.025 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-01T04:16:51.108 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:51.107+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:51.142 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:51.142 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:51.156 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-01T04:16:51.208 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:51.207+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:51.279 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:51.279 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:51.293 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-01T04:16:51.309 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:51.307+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:51.410 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:51.409+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:51.417 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:51.417 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:51.431 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-01T04:16:51.510 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:51.508+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:51.549 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:51.549 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:51.563 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-01T04:16:51.613 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:51.607+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:51.712 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:51.711+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:51.732 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:51.732 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:51.746 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-01T04:16:51.777 INFO:teuthology.orchestra.run.smithi153.stderr:nodeep-scrub is unset 2023-11-01T04:16:51.785 INFO:tasks.rados.rados.0.smithi190.stdout:241: done (8 left) 2023-11-01T04:16:51.785 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 26 current snap is 21 2023-11-01T04:16:51.785 INFO:tasks.rados.rados.0.smithi190.stdout:242: seq_num 99 ranges {425984=507904,933888=606208,1540096=557056} 2023-11-01T04:16:51.792 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2023-11-01T04:16:51.792 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-01T04:16:51.792 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2023-11-01T04:16:51.792 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2023-11-01T04:16:51.799 INFO:tasks.rados.rados.0.smithi190.stdout:242: writing smithi19039307-26 from 425984 to 933888 tid 1 2023-11-01T04:16:51.806 INFO:tasks.rados.rados.0.smithi190.stdout:242: writing smithi19039307-26 from 933888 to 1540096 tid 2 2023-11-01T04:16:51.814 INFO:tasks.rados.rados.0.smithi190.stdout:242: writing smithi19039307-26 from 1540096 to 2097152 tid 3 2023-11-01T04:16:51.814 INFO:tasks.rados.rados.0.smithi190.stdout:243: copy_from oid 33 from oid 48 current snap is 21 2023-11-01T04:16:51.814 INFO:tasks.rados.rados.0.smithi190.stdout:231: expect (ObjNum 33 snap 0 seq_num 33) 2023-11-01T04:16:51.843 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:51.844+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:51.857 INFO:tasks.rados.rados.0.smithi190.stdout:230: finishing write tid 5 to smithi19039307-14 2023-11-01T04:16:51.857 INFO:tasks.rados.rados.0.smithi190.stdout:230: oid 14 updating version 0 to 63 2023-11-01T04:16:51.857 INFO:tasks.rados.rados.0.smithi190.stdout:230: oid 14 updating version 63 to 65 2023-11-01T04:16:51.857 INFO:tasks.rados.rados.0.smithi190.stdout:230: oid 14 updating version 65 to 66 2023-11-01T04:16:51.858 INFO:tasks.rados.rados.0.smithi190.stdout:230: oid 14 version 66 is already newer than 64 2023-11-01T04:16:51.858 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 14 v 66 (ObjNum 94 snap 21 seq_num 94) dirty exists 2023-11-01T04:16:51.858 INFO:tasks.rados.rados.0.smithi190.stdout:230: left oid 14 (ObjNum 94 snap 21 seq_num 94) 2023-11-01T04:16:51.858 INFO:tasks.rados.rados.0.smithi190.stdout:237: finishing rollback tid 0 to smithi19039307-27 2023-11-01T04:16:51.858 INFO:tasks.rados.rados.0.smithi190.stdout:239: finishing rollback tid 0 to smithi19039307-25 2023-11-01T04:16:51.859 INFO:tasks.rados.rados.0.smithi190.stdout:239: finishing rollback tid 1 to smithi19039307-25 2023-11-01T04:16:51.859 INFO:tasks.rados.rados.0.smithi190.stdout:232: finishing write tid 4 to smithi19039307-24 2023-11-01T04:16:51.859 INFO:tasks.rados.rados.0.smithi190.stdout:239: finishing rollback tid 2 to smithi19039307-25 2023-11-01T04:16:51.859 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 25 v 50 (ObjNum 76 snap 13 seq_num 76) dirty exists 2023-11-01T04:16:51.859 INFO:tasks.rados.rados.0.smithi190.stdout:237: finishing rollback tid 1 to smithi19039307-27 2023-11-01T04:16:51.860 INFO:tasks.rados.rados.0.smithi190.stdout:237: finishing rollback tid 2 to smithi19039307-27 2023-11-01T04:16:51.860 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 27 v 78 (ObjNum 66 snap 6 seq_num 66) dirty exists 2023-11-01T04:16:51.860 INFO:tasks.rados.rados.0.smithi190.stdout:234: finishing rollback tid 0 to smithi19039307-49 2023-11-01T04:16:51.860 INFO:tasks.rados.rados.0.smithi190.stdout:232: finishing write tid 5 to smithi19039307-24 2023-11-01T04:16:51.860 INFO:tasks.rados.rados.0.smithi190.stdout:232: oid 24 updating version 0 to 79 2023-11-01T04:16:51.860 INFO:tasks.rados.rados.0.smithi190.stdout:232: oid 24 version 79 is already newer than 78 2023-11-01T04:16:51.861 INFO:tasks.rados.rados.0.smithi190.stdout:232: oid 24 version 79 is already newer than 77 2023-11-01T04:16:51.861 INFO:tasks.rados.rados.0.smithi190.stdout:232: oid 24 updating version 79 to 80 2023-11-01T04:16:51.861 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 24 v 80 (ObjNum 95 snap 21 seq_num 95) dirty exists 2023-11-01T04:16:51.861 INFO:tasks.rados.rados.0.smithi190.stdout:232: left oid 24 (ObjNum 95 snap 21 seq_num 95) 2023-11-01T04:16:51.861 INFO:tasks.rados.rados.0.smithi190.stdout:236: finishing copy_from racing read to smithi19039307-36 2023-11-01T04:16:51.862 INFO:tasks.rados.rados.0.smithi190.stdout:230: done (9 left) 2023-11-01T04:16:51.862 INFO:tasks.rados.rados.0.smithi190.stdout:231: done (8 left) 2023-11-01T04:16:51.862 INFO:tasks.rados.rados.0.smithi190.stdout:232: done (7 left) 2023-11-01T04:16:51.862 INFO:tasks.rados.rados.0.smithi190.stdout:237: done (6 left) 2023-11-01T04:16:51.862 INFO:tasks.rados.rados.0.smithi190.stdout:239: done (5 left) 2023-11-01T04:16:51.862 INFO:tasks.rados.rados.0.smithi190.stdout:244: delete oid 23 current snap is 21 2023-11-01T04:16:51.863 INFO:tasks.rados.rados.0.smithi190.stdout:234: finishing rollback tid 1 to smithi19039307-49 2023-11-01T04:16:51.863 INFO:tasks.rados.rados.0.smithi190.stdout:234: finishing rollback tid 2 to smithi19039307-49 2023-11-01T04:16:51.863 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 49 v 72 (ObjNum 60 snap 5 seq_num 60) dirty exists 2023-11-01T04:16:51.863 INFO:tasks.rados.rados.0.smithi190.stdout:238: finishing write tid 1 to smithi19039307-13 2023-11-01T04:16:51.863 INFO:tasks.rados.rados.0.smithi190.stdout:238: finishing write tid 2 to smithi19039307-13 2023-11-01T04:16:51.864 INFO:tasks.rados.rados.0.smithi190.stdout:238: finishing write tid 3 to smithi19039307-13 2023-11-01T04:16:51.864 INFO:tasks.rados.rados.0.smithi190.stdout:238: finishing write tid 4 to smithi19039307-13 2023-11-01T04:16:51.864 INFO:tasks.rados.rados.0.smithi190.stdout:238: finishing write tid 5 to smithi19039307-13 2023-11-01T04:16:51.864 INFO:tasks.rados.rados.0.smithi190.stdout:236: finishing copy_from to smithi19039307-36 2023-11-01T04:16:51.864 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 36 v 39 (ObjNum 53 snap 1 seq_num 53) dirty exists 2023-11-01T04:16:51.865 INFO:tasks.rados.rados.0.smithi190.stdout:238: finishing write tid 6 to smithi19039307-13 2023-11-01T04:16:51.865 INFO:tasks.rados.rados.0.smithi190.stdout:238: oid 13 updating version 0 to 75 2023-11-01T04:16:51.865 INFO:tasks.rados.rados.0.smithi190.stdout:238: oid 13 version 75 is already newer than 74 2023-11-01T04:16:51.865 INFO:tasks.rados.rados.0.smithi190.stdout:238: oid 13 updating version 75 to 78 2023-11-01T04:16:51.865 INFO:tasks.rados.rados.0.smithi190.stdout:238: oid 13 version 78 is already newer than 76 2023-11-01T04:16:51.866 INFO:tasks.rados.rados.0.smithi190.stdout:238: oid 13 version 78 is already newer than 77 2023-11-01T04:16:51.866 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 13 v 78 (ObjNum 98 snap 21 seq_num 98) dirty exists 2023-11-01T04:16:51.866 INFO:tasks.rados.rados.0.smithi190.stdout:238: left oid 13 (ObjNum 98 snap 21 seq_num 98) 2023-11-01T04:16:51.871 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:51.871 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:51.886 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-01T04:16:51.914 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:51.910+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:52.015 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:52.010+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:52.043 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:52.044 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:52.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-01T04:16:52.063 INFO:tasks.rados.rados.0.smithi190.stdout:234: done (5 left) 2023-11-01T04:16:52.063 INFO:tasks.rados.rados.0.smithi190.stdout:236: done (4 left) 2023-11-01T04:16:52.063 INFO:tasks.rados.rados.0.smithi190.stdout:238: done (3 left) 2023-11-01T04:16:52.063 INFO:tasks.rados.rados.0.smithi190.stdout:244: done (2 left) 2023-11-01T04:16:52.064 INFO:tasks.rados.rados.0.smithi190.stdout:245: delete oid 40 current snap is 21 2023-11-01T04:16:52.097 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-11-01T04:16:52.096+0000 7f8f9f6a2640 -1 mon.a@0(leader).osd e71 definitely_dead 0 2023-11-01T04:16:52.115 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:52.110+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:52.158 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:16:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:16:52.161 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-01T04:16:52.247 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:52.248+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:52.285 INFO:tasks.rados.rados.0.smithi190.stdout:243: finishing copy_from racing read to smithi19039307-33 2023-11-01T04:16:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:16:52.314 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:16:52.315 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-01T04:16:52.317 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:52.317+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:52.319 INFO:tasks.rados.rados.0.smithi190.stdout:242: finishing write tid 1 to smithi19039307-26 2023-11-01T04:16:52.320 INFO:tasks.rados.rados.0.smithi190.stdout:242: finishing write tid 2 to smithi19039307-26 2023-11-01T04:16:52.417 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:16:52.459 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:16:52.462 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:16:52.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-01T04:16:52.505 INFO:tasks.rados.rados.0.smithi190.stdout:245: done (2 left) 2023-11-01T04:16:52.505 INFO:tasks.rados.rados.0.smithi190.stdout:246: setattr oid 8 current snap is 21 2023-11-01T04:16:52.511 INFO:tasks.rados.rados.0.smithi190.stdout:247: snap_remove snap 2 2023-11-01T04:16:52.519 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:52.517+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:52.594 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:52.594 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:52.608 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-01T04:16:52.619 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:16:52.711 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-01T04:16:52.721 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:52.717+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:52.767 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:52.767 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:52.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-01T04:16:52.787 INFO:teuthology.orchestra.run.smithi153.stderr:marked out osd.5. 2023-11-01T04:16:52.794 INFO:tasks.rados.rados.0.smithi190.stdout:247: done (3 left) 2023-11-01T04:16:52.794 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 44 current snap is 21 2023-11-01T04:16:52.794 INFO:tasks.rados.rados.0.smithi190.stdout:243: finishing copy_from to smithi19039307-33 2023-11-01T04:16:52.795 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 33 v 23 (ObjNum 71 snap 8 seq_num 71) dirty exists 2023-11-01T04:16:52.795 INFO:tasks.rados.rados.0.smithi190.stdout:248: seq_num 102 ranges {0=557056,557056=638976,1196032=524288} 2023-11-01T04:16:52.810 INFO:tasks.rados.rados.0.smithi190.stdout:248: writing smithi19039307-44 from 0 to 557056 tid 1 2023-11-01T04:16:52.821 INFO:tasks.rados.rados.0.smithi190.stdout:248: writing smithi19039307-44 from 557056 to 1196032 tid 2 2023-11-01T04:16:52.825 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:52.825+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:52.830 INFO:tasks.rados.rados.0.smithi190.stdout:248: writing smithi19039307-44 from 1196032 to 1720320 tid 3 2023-11-01T04:16:52.830 INFO:tasks.rados.rados.0.smithi190.stdout:243: done (3 left) 2023-11-01T04:16:52.831 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 5 current snap is 21 2023-11-01T04:16:52.831 INFO:tasks.rados.rados.0.smithi190.stdout:249: seq_num 103 ranges {81920=573440,655360=786432,1441792=376832} 2023-11-01T04:16:52.841 INFO:tasks.rados.rados.0.smithi190.stdout:249: writing smithi19039307-5 from 81920 to 655360 tid 1 2023-11-01T04:16:52.852 INFO:tasks.rados.rados.0.smithi190.stdout:249: writing smithi19039307-5 from 655360 to 1441792 tid 2 2023-11-01T04:16:52.856 INFO:tasks.rados.rados.0.smithi190.stdout:249: writing smithi19039307-5 from 1441792 to 1818624 tid 3 2023-11-01T04:16:52.857 INFO:tasks.rados.rados.0.smithi190.stdout:250: copy_from oid 48 from oid 39 current snap is 21 2023-11-01T04:16:52.857 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 8 v 82 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-11-01T04:16:52.857 INFO:tasks.rados.rados.0.smithi190.stdout:242: finishing write tid 3 to smithi19039307-26 2023-11-01T04:16:52.857 INFO:tasks.rados.rados.0.smithi190.stdout:242: finishing write tid 4 to smithi19039307-26 2023-11-01T04:16:52.857 INFO:tasks.rados.rados.0.smithi190.stdout:246: done (4 left) 2023-11-01T04:16:52.858 INFO:tasks.rados.rados.0.smithi190.stdout:251: snap_create 2023-11-01T04:16:52.872 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:52.873 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-11-01T04:16:52.922 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:52.921+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:52.925 INFO:tasks.rados.rados.0.smithi190.stdout:242: finishing write tid 5 to smithi19039307-26 2023-11-01T04:16:52.925 INFO:tasks.rados.rados.0.smithi190.stdout:242: oid 26 updating version 0 to 24 2023-11-01T04:16:52.925 INFO:tasks.rados.rados.0.smithi190.stdout:242: oid 26 version 24 is already newer than 23 2023-11-01T04:16:52.925 INFO:tasks.rados.rados.0.smithi190.stdout:242: oid 26 updating version 24 to 25 2023-11-01T04:16:52.925 INFO:tasks.rados.rados.0.smithi190.stdout:242: oid 26 updating version 25 to 26 2023-11-01T04:16:52.926 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 26 v 26 (ObjNum 99 snap 21 seq_num 99) dirty exists 2023-11-01T04:16:52.926 INFO:tasks.rados.rados.0.smithi190.stdout:242: left oid 26 (ObjNum 99 snap 21 seq_num 99) 2023-11-01T04:16:52.932 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:52.933 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-01T04:16:52.954 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:52.954 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:16:52.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:16:52.954 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:16:52.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.955 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/31 ECSubWrite(tid=18, reqid=client.4559.0:125, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:52.955 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.281930+0000", 2023-11-01T04:16:52.955 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 37.670320736999997, 2023-11-01T04:16:52.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20154303100000001, 2023-11-01T04:16:52.956 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:52.956 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:52.956 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:52.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:52.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281930+0000", 2023-11-01T04:16:52.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:52.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:52.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281930+0000", 2023-11-01T04:16:52.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:52.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:52.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281931+0000", 2023-11-01T04:16:52.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6400000000000001e-07 2023-11-01T04:16:52.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:52.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281961+0000", 2023-11-01T04:16:52.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9621999999999999e-05 2023-11-01T04:16:52.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:52.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281987+0000", 2023-11-01T04:16:52.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6316e-05 2023-11-01T04:16:52.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:52.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382103+0000", 2023-11-01T04:16:52.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100116649 2023-11-01T04:16:52.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:52.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.482335+0000", 2023-11-01T04:16:52.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100231813 2023-11-01T04:16:52.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:52.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.482364+0000", 2023-11-01T04:16:52.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8829000000000001e-05 2023-11-01T04:16:52.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:52.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.483427+0000", 2023-11-01T04:16:52.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010627970000000001 2023-11-01T04:16:52.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:52.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.483473+0000", 2023-11-01T04:16:52.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6241e-05 2023-11-01T04:16:52.965 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:52.966 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:52.966 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:52.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.966 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/31 ECSubWrite(tid=4, reqid=client.4559.0:136, at_version=36'3, trim_to=0'0, roll_forward_to=33'2))", 2023-11-01T04:16:52.966 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.284706+0000", 2023-11-01T04:16:52.967 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 37.667544829000001, 2023-11-01T04:16:52.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202677317, 2023-11-01T04:16:52.967 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:52.967 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:52.967 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:52.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:52.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.284706+0000", 2023-11-01T04:16:52.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:52.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:52.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.284706+0000", 2023-11-01T04:16:52.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:52.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:52.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.284707+0000", 2023-11-01T04:16:52.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8599999999999998e-07 2023-11-01T04:16:52.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:52.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.285485+0000", 2023-11-01T04:16:52.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077885900000000002 2023-11-01T04:16:52.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:52.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.285511+0000", 2023-11-01T04:16:52.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5378999999999999e-05 2023-11-01T04:16:52.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:52.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385600+0000", 2023-11-01T04:16:52.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100089631 2023-11-01T04:16:52.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:52.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385749+0000", 2023-11-01T04:16:52.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014876399999999999 2023-11-01T04:16:52.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:52.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385771+0000", 2023-11-01T04:16:52.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1535e-05 2023-11-01T04:16:52.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:52.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.487315+0000", 2023-11-01T04:16:52.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101544735 2023-11-01T04:16:52.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:52.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.487383+0000", 2023-11-01T04:16:52.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7928000000000004e-05 2023-11-01T04:16:52.977 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:52.977 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:52.977 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:52.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.978 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 36/31 ECSubWrite(tid=14, reqid=client.4559.0:146, at_version=36'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:52.978 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.322569+0000", 2023-11-01T04:16:52.978 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 37.629681935000001, 2023-11-01T04:16:52.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20180657299999999, 2023-11-01T04:16:52.978 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:52.979 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:52.979 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:52.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:52.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322569+0000", 2023-11-01T04:16:52.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:52.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:52.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322569+0000", 2023-11-01T04:16:52.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:52.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:52.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322571+0000", 2023-11-01T04:16:52.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8029999999999999e-06 2023-11-01T04:16:52.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:52.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322759+0000", 2023-11-01T04:16:52.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018862300000000001 2023-11-01T04:16:52.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:52.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322790+0000", 2023-11-01T04:16:52.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0712e-05 2023-11-01T04:16:52.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:52.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322807+0000", 2023-11-01T04:16:52.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6566e-05 2023-11-01T04:16:52.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:52.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.423031+0000", 2023-11-01T04:16:52.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10022463600000001 2023-11-01T04:16:52.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:52.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.423046+0000", 2023-11-01T04:16:52.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4632e-05 2023-11-01T04:16:52.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:52.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524322+0000", 2023-11-01T04:16:52.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101275743 2023-11-01T04:16:52.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:52.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524376+0000", 2023-11-01T04:16:52.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3857999999999998e-05 2023-11-01T04:16:52.988 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:52.988 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:52.988 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:52.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.989 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 36/31 ECSubWrite(tid=16, reqid=client.4559.0:147, at_version=36'10, trim_to=0'0, roll_forward_to=36'8))", 2023-11-01T04:16:52.989 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.382654+0000", 2023-11-01T04:16:52.989 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 37.569597334000001, 2023-11-01T04:16:52.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14175697400000001, 2023-11-01T04:16:52.989 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:52.989 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:52.990 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:52.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:52.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382654+0000", 2023-11-01T04:16:52.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:52.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:52.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382654+0000", 2023-11-01T04:16:52.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:52.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:52.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382655+0000", 2023-11-01T04:16:52.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.933e-06 2023-11-01T04:16:52.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:52.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382823+0000", 2023-11-01T04:16:52.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000167806 2023-11-01T04:16:52.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:52.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382856+0000", 2023-11-01T04:16:52.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2255999999999997e-05 2023-11-01T04:16:52.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:52.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382926+0000", 2023-11-01T04:16:52.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0759999999999993e-05 2023-11-01T04:16:52.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:52.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.424403+0000", 2023-11-01T04:16:52.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.041477075000000002 2023-11-01T04:16:52.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:52.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.424416+0000", 2023-11-01T04:16:52.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2249000000000001e-05 2023-11-01T04:16:52.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:52.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524381+0000", 2023-11-01T04:16:52.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.099965719999999994 2023-11-01T04:16:52.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:52.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:52.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:52.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524411+0000", 2023-11-01T04:16:52.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9175000000000001e-05 2023-11-01T04:16:52.999 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:52.999 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:52.999 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:52.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.000 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 36/31 ECSubWrite(tid=17, reqid=client.4559.0:148, at_version=36'11, trim_to=0'0, roll_forward_to=36'8))", 2023-11-01T04:16:53.000 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.385040+0000", 2023-11-01T04:16:53.000 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 37.567210871, 2023-11-01T04:16:53.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14044638700000001, 2023-11-01T04:16:53.000 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.001 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.001 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385040+0000", 2023-11-01T04:16:53.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385040+0000", 2023-11-01T04:16:53.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385042+0000", 2023-11-01T04:16:53.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7129999999999999e-06 2023-11-01T04:16:53.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385087+0000", 2023-11-01T04:16:53.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4895000000000003e-05 2023-11-01T04:16:53.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385121+0000", 2023-11-01T04:16:53.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4180999999999997e-05 2023-11-01T04:16:53.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385145+0000", 2023-11-01T04:16:53.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3949000000000001e-05 2023-11-01T04:16:53.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.523994+0000", 2023-11-01T04:16:53.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13884963 2023-11-01T04:16:53.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524008+0000", 2023-11-01T04:16:53.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3719e-05 2023-11-01T04:16:53.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.525407+0000", 2023-11-01T04:16:53.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013987229999999999 2023-11-01T04:16:53.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.525486+0000", 2023-11-01T04:16:53.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9577000000000001e-05 2023-11-01T04:16:53.010 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.010 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.010 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.011 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 37/31 ECSubWrite(tid=22, reqid=client.4559.0:197, at_version=37'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:53.011 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:16:53.011 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.911979322999997, 2023-11-01T04:16:53.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.301677274, 2023-11-01T04:16:53.012 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.012 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.012 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:16:53.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:16:53.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040273+0000", 2023-11-01T04:16:53.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3370000000000001e-06 2023-11-01T04:16:53.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040307+0000", 2023-11-01T04:16:53.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3714999999999998e-05 2023-11-01T04:16:53.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040331+0000", 2023-11-01T04:16:53.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4409e-05 2023-11-01T04:16:53.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140479+0000", 2023-11-01T04:16:53.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10014764900000001 2023-11-01T04:16:53.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.240734+0000", 2023-11-01T04:16:53.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100255645 2023-11-01T04:16:53.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.240751+0000", 2023-11-01T04:16:53.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6852000000000001e-05 2023-11-01T04:16:53.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341894+0000", 2023-11-01T04:16:53.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101143302 2023-11-01T04:16:53.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341949+0000", 2023-11-01T04:16:53.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4364999999999999e-05 2023-11-01T04:16:53.021 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.021 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.022 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.022 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 37/31 ECSubWrite(tid=23, reqid=client.4559.0:198, at_version=37'10, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:53.022 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.140534+0000", 2023-11-01T04:16:53.023 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.811716429000001, 2023-11-01T04:16:53.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202392451, 2023-11-01T04:16:53.023 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.023 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.023 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140534+0000", 2023-11-01T04:16:53.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140534+0000", 2023-11-01T04:16:53.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140536+0000", 2023-11-01T04:16:53.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.22e-06 2023-11-01T04:16:53.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140578+0000", 2023-11-01T04:16:53.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2715e-05 2023-11-01T04:16:53.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140603+0000", 2023-11-01T04:16:53.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4609000000000001e-05 2023-11-01T04:16:53.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140616+0000", 2023-11-01T04:16:53.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2763e-05 2023-11-01T04:16:53.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341493+0000", 2023-11-01T04:16:53.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.200877214 2023-11-01T04:16:53.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341508+0000", 2023-11-01T04:16:53.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5013e-05 2023-11-01T04:16:53.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.342869+0000", 2023-11-01T04:16:53.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001361183 2023-11-01T04:16:53.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.342927+0000", 2023-11-01T04:16:53.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7734e-05 2023-11-01T04:16:53.032 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.032 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.033 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.033 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 37/31 ECSubWrite(tid=13, reqid=client.4559.0:181, at_version=37'8, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:53.033 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.248764+0000", 2023-11-01T04:16:53.034 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.703487043999999, 2023-11-01T04:16:53.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20321630399999999, 2023-11-01T04:16:53.034 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.034 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.034 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.248764+0000", 2023-11-01T04:16:53.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.248764+0000", 2023-11-01T04:16:53.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.248766+0000", 2023-11-01T04:16:53.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8300000000000001e-06 2023-11-01T04:16:53.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.249255+0000", 2023-11-01T04:16:53.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00048892399999999996 2023-11-01T04:16:53.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.249286+0000", 2023-11-01T04:16:53.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1004999999999999e-05 2023-11-01T04:16:53.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.249302+0000", 2023-11-01T04:16:53.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6169999999999999e-05 2023-11-01T04:16:53.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.349516+0000", 2023-11-01T04:16:53.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10021408599999999 2023-11-01T04:16:53.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.349530+0000", 2023-11-01T04:16:53.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4472e-05 2023-11-01T04:16:53.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451931+0000", 2023-11-01T04:16:53.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10240034100000001 2023-11-01T04:16:53.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451980+0000", 2023-11-01T04:16:53.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9475999999999999e-05 2023-11-01T04:16:53.043 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.044 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.044 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.044 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 37/31 ECSubWrite(tid=14, reqid=client.4559.0:182, at_version=37'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:53.044 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.253221+0000", 2023-11-01T04:16:53.045 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.699029887999998, 2023-11-01T04:16:53.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19911255999999999, 2023-11-01T04:16:53.045 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.045 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.045 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253221+0000", 2023-11-01T04:16:53.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253221+0000", 2023-11-01T04:16:53.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253222+0000", 2023-11-01T04:16:53.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1570000000000001e-06 2023-11-01T04:16:53.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253418+0000", 2023-11-01T04:16:53.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019597500000000001 2023-11-01T04:16:53.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253443+0000", 2023-11-01T04:16:53.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5055999999999999e-05 2023-11-01T04:16:53.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253459+0000", 2023-11-01T04:16:53.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5495999999999999e-05 2023-11-01T04:16:53.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451029+0000", 2023-11-01T04:16:53.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19757031799999999 2023-11-01T04:16:53.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451039+0000", 2023-11-01T04:16:53.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0062e-05 2023-11-01T04:16:53.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.452298+0000", 2023-11-01T04:16:53.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012584849999999999 2023-11-01T04:16:53.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.452334+0000", 2023-11-01T04:16:53.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6010999999999998e-05 2023-11-01T04:16:53.055 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.055 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.055 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.055 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 37/31 ECSubWrite(tid=15, reqid=client.4559.0:183, at_version=37'10, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:16:53.056 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.255859+0000", 2023-11-01T04:16:53.056 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.696392195999998, 2023-11-01T04:16:53.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19868693900000001, 2023-11-01T04:16:53.056 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.056 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.056 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255859+0000", 2023-11-01T04:16:53.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255859+0000", 2023-11-01T04:16:53.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255860+0000", 2023-11-01T04:16:53.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2300000000000001e-06 2023-11-01T04:16:53.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255894+0000", 2023-11-01T04:16:53.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3796e-05 2023-11-01T04:16:53.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255923+0000", 2023-11-01T04:16:53.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9352e-05 2023-11-01T04:16:53.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255936+0000", 2023-11-01T04:16:53.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3232e-05 2023-11-01T04:16:53.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451650+0000", 2023-11-01T04:16:53.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19571368 2023-11-01T04:16:53.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451665+0000", 2023-11-01T04:16:53.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5119e-05 2023-11-01T04:16:53.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.454503+0000", 2023-11-01T04:16:53.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028377400000000001 2023-11-01T04:16:53.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.454546+0000", 2023-11-01T04:16:53.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2790000000000002e-05 2023-11-01T04:16:53.066 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.066 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.066 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.067 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpRead(3.es2 41/31 ECSubRead(tid=31, to_read={3:706dddd5:::smithi19039307-14:head=0,159744,0}, subchunks={3:706dddd5:::smithi19039307-14:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:53.067 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:21.022133+0000", 2023-11-01T04:16:53.067 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 31.930118187000001, 2023-11-01T04:16:53.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16269023099999999, 2023-11-01T04:16:53.067 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.067 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.068 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022133+0000", 2023-11-01T04:16:53.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022133+0000", 2023-11-01T04:16:53.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022133+0000", 2023-11-01T04:16:53.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2e-07 2023-11-01T04:16:53.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022144+0000", 2023-11-01T04:16:53.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.118e-05 2023-11-01T04:16:53.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022175+0000", 2023-11-01T04:16:53.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0518999999999997e-05 2023-11-01T04:16:53.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.022204+0000", 2023-11-01T04:16:53.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8878999999999999e-05 2023-11-01T04:16:53.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.184034+0000", 2023-11-01T04:16:53.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.161829903 2023-11-01T04:16:53.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.184823+0000", 2023-11-01T04:16:53.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00078932999999999998 2023-11-01T04:16:53.075 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.075 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.075 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.076 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpRead(3.es2 41/31 ECSubRead(tid=32, to_read={3:706dddd5:::smithi19039307-14:head=0,159744,0}, subchunks={3:706dddd5:::smithi19039307-14:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:53.076 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:21.023180+0000", 2023-11-01T04:16:53.076 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 31.92907052, 2023-11-01T04:16:53.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16201669099999999, 2023-11-01T04:16:53.077 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.077 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.077 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023180+0000", 2023-11-01T04:16:53.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023180+0000", 2023-11-01T04:16:53.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023182+0000", 2023-11-01T04:16:53.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0470000000000001e-06 2023-11-01T04:16:53.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023203+0000", 2023-11-01T04:16:53.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0845000000000001e-05 2023-11-01T04:16:53.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023244+0000", 2023-11-01T04:16:53.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1118999999999997e-05 2023-11-01T04:16:53.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.023280+0000", 2023-11-01T04:16:53.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5812000000000002e-05 2023-11-01T04:16:53.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.184966+0000", 2023-11-01T04:16:53.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16168543899999999 2023-11-01T04:16:53.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.185197+0000", 2023-11-01T04:16:53.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023142900000000001 2023-11-01T04:16:53.084 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.084 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.085 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.085 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 48/31 ECSubWrite(tid=54, reqid=client.4559.0:369, at_version=48'22, trim_to=0'0, roll_forward_to=48'20))", 2023-11-01T04:16:53.085 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:28.168501+0000", 2023-11-01T04:16:53.086 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 24.783749582999999, 2023-11-01T04:16:53.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.198147297, 2023-11-01T04:16:53.086 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.086 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.086 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.168501+0000", 2023-11-01T04:16:53.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.168501+0000", 2023-11-01T04:16:53.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.168503+0000", 2023-11-01T04:16:53.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.55e-06 2023-11-01T04:16:53.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.168567+0000", 2023-11-01T04:16:53.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4234999999999997e-05 2023-11-01T04:16:53.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.168603+0000", 2023-11-01T04:16:53.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5536e-05 2023-11-01T04:16:53.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.168625+0000", 2023-11-01T04:16:53.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2671999999999999e-05 2023-11-01T04:16:53.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.365641+0000", 2023-11-01T04:16:53.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.197015577 2023-11-01T04:16:53.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.365647+0000", 2023-11-01T04:16:53.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3049999999999998e-06 2023-11-01T04:16:53.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.366616+0000", 2023-11-01T04:16:53.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00096908400000000005 2023-11-01T04:16:53.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.366649+0000", 2023-11-01T04:16:53.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2338000000000001e-05 2023-11-01T04:16:53.098 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.098 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.098 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.099 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 48/31 ECSubWrite(tid=55, reqid=client.4559.0:370, at_version=48'23, trim_to=0'0, roll_forward_to=48'20))", 2023-11-01T04:16:53.099 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:28.170472+0000", 2023-11-01T04:16:53.099 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 24.781778865, 2023-11-01T04:16:53.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19697836399999999, 2023-11-01T04:16:53.100 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.100 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.101 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.170472+0000", 2023-11-01T04:16:53.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.170472+0000", 2023-11-01T04:16:53.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.170473+0000", 2023-11-01T04:16:53.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1370000000000001e-06 2023-11-01T04:16:53.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.170502+0000", 2023-11-01T04:16:53.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8642000000000001e-05 2023-11-01T04:16:53.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.170525+0000", 2023-11-01T04:16:53.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3349999999999998e-05 2023-11-01T04:16:53.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.270635+0000", 2023-11-01T04:16:53.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100109552 2023-11-01T04:16:53.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.366031+0000", 2023-11-01T04:16:53.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.095395876000000004 2023-11-01T04:16:53.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.366038+0000", 2023-11-01T04:16:53.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0859999999999996e-06 2023-11-01T04:16:53.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.367415+0000", 2023-11-01T04:16:53.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013778429999999999 2023-11-01T04:16:53.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.367450+0000", 2023-11-01T04:16:53.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4878e-05 2023-11-01T04:16:53.110 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.110 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.110 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.111 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 58/31 ECSubWrite(tid=63, reqid=osd.7.0:63, at_version=58'57, trim_to=0'0, roll_forward_to=57'53))", 2023-11-01T04:16:53.111 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:38.447186+0000", 2023-11-01T04:16:53.111 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 14.505064974, 2023-11-01T04:16:53.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202892298, 2023-11-01T04:16:53.112 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.112 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.112 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447186+0000", 2023-11-01T04:16:53.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447186+0000", 2023-11-01T04:16:53.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447187+0000", 2023-11-01T04:16:53.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.333e-06 2023-11-01T04:16:53.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447243+0000", 2023-11-01T04:16:53.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5461999999999999e-05 2023-11-01T04:16:53.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447268+0000", 2023-11-01T04:16:53.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5286000000000002e-05 2023-11-01T04:16:53.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447289+0000", 2023-11-01T04:16:53.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1060000000000002e-05 2023-11-01T04:16:53.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.647605+0000", 2023-11-01T04:16:53.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20031611899999999 2023-11-01T04:16:53.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.647619+0000", 2023-11-01T04:16:53.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4316e-05 2023-11-01T04:16:53.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.650003+0000", 2023-11-01T04:16:53.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002383994 2023-11-01T04:16:53.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.650078+0000", 2023-11-01T04:16:53.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4727999999999993e-05 2023-11-01T04:16:53.121 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.121 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.121 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.122 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.bs2 62/31 ECSubWrite(tid=70, reqid=client.4559.0:479, at_version=62'17, trim_to=0'0, roll_forward_to=45'15))", 2023-11-01T04:16:53.122 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:42.297514+0000", 2023-11-01T04:16:53.122 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.654736740000001, 2023-11-01T04:16:53.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10530524099999999, 2023-11-01T04:16:53.123 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.123 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.123 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.297514+0000", 2023-11-01T04:16:53.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.297514+0000", 2023-11-01T04:16:53.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.297515+0000", 2023-11-01T04:16:53.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1300000000000001e-07 2023-11-01T04:16:53.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.298129+0000", 2023-11-01T04:16:53.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00061414599999999998 2023-11-01T04:16:53.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.298189+0000", 2023-11-01T04:16:53.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0109000000000003e-05 2023-11-01T04:16:53.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.298216+0000", 2023-11-01T04:16:53.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6903e-05 2023-11-01T04:16:53.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.398468+0000", 2023-11-01T04:16:53.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100251969 2023-11-01T04:16:53.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.398482+0000", 2023-11-01T04:16:53.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4749999999999999e-05 2023-11-01T04:16:53.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.402749+0000", 2023-11-01T04:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042666229999999998 2023-11-01T04:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.402819+0000", 2023-11-01T04:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0327999999999995e-05 2023-11-01T04:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.133 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.0s3 67/31 ECSubWrite(tid=35, reqid=osd.3.0:34, at_version=0'0, trim_to=0'0, roll_forward_to=67'19))", 2023-11-01T04:16:53.133 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.288148+0000", 2023-11-01T04:16:53.133 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.6641028809999998, 2023-11-01T04:16:53.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20173064900000001, 2023-11-01T04:16:53.133 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.133 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.134 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.288148+0000", 2023-11-01T04:16:53.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.288148+0000", 2023-11-01T04:16:53.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.288150+0000", 2023-11-01T04:16:53.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4839999999999998e-06 2023-11-01T04:16:53.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.288174+0000", 2023-11-01T04:16:53.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3879999999999998e-05 2023-11-01T04:16:53.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.288206+0000", 2023-11-01T04:16:53.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1668999999999998e-05 2023-11-01T04:16:53.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.388354+0000", 2023-11-01T04:16:53.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10014759600000001 2023-11-01T04:16:53.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.488597+0000", 2023-11-01T04:16:53.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100242922 2023-11-01T04:16:53.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.488623+0000", 2023-11-01T04:16:53.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6713e-05 2023-11-01T04:16:53.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.489797+0000", 2023-11-01T04:16:53.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001174034 2023-11-01T04:16:53.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.489879+0000", 2023-11-01T04:16:53.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1351000000000001e-05 2023-11-01T04:16:53.143 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.143 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.143 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.144 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 67/31 ECSubWrite(tid=77, reqid=client.4559.0:537, at_version=67'58, trim_to=0'0, roll_forward_to=67'56))", 2023-11-01T04:16:53.144 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.705633+0000", 2023-11-01T04:16:53.144 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.2466175929999999, 2023-11-01T04:16:53.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19831848199999999, 2023-11-01T04:16:53.144 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.144 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.145 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.705633+0000", 2023-11-01T04:16:53.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.705633+0000", 2023-11-01T04:16:53.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.705635+0000", 2023-11-01T04:16:53.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.793e-06 2023-11-01T04:16:53.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.706212+0000", 2023-11-01T04:16:53.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00057694199999999999 2023-11-01T04:16:53.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.706243+0000", 2023-11-01T04:16:53.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0593000000000003e-05 2023-11-01T04:16:53.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.706258+0000", 2023-11-01T04:16:53.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5556000000000001e-05 2023-11-01T04:16:53.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.802680+0000", 2023-11-01T04:16:53.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.096422161000000006 2023-11-01T04:16:53.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.802699+0000", 2023-11-01T04:16:53.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9023999999999999e-05 2023-11-01T04:16:53.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.903896+0000", 2023-11-01T04:16:53.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10119634499999999 2023-11-01T04:16:53.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.903952+0000", 2023-11-01T04:16:53.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6067999999999999e-05 2023-11-01T04:16:53.154 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.154 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.154 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.154 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.2s5 69/31 ECSubWrite(tid=323, reqid=client.4559.0:568, at_version=69'77, trim_to=0'0, roll_forward_to=68'74))", 2023-11-01T04:16:53.155 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:49.538847+0000", 2023-11-01T04:16:53.155 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.413404002, 2023-11-01T04:16:53.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.107648559, 2023-11-01T04:16:53.155 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.155 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.156 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.538847+0000", 2023-11-01T04:16:53.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.538847+0000", 2023-11-01T04:16:53.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.538849+0000", 2023-11-01T04:16:53.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1040000000000001e-06 2023-11-01T04:16:53.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.539027+0000", 2023-11-01T04:16:53.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017823599999999999 2023-11-01T04:16:53.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.539062+0000", 2023-11-01T04:16:53.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5196e-05 2023-11-01T04:16:53.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.539079+0000", 2023-11-01T04:16:53.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6742999999999999e-05 2023-11-01T04:16:53.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.639368+0000", 2023-11-01T04:16:53.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10028886300000001 2023-11-01T04:16:53.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.639399+0000", 2023-11-01T04:16:53.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1296e-05 2023-11-01T04:16:53.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.646434+0000", 2023-11-01T04:16:53.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0070349690000000003 2023-11-01T04:16:53.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.646495+0000", 2023-11-01T04:16:53.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1152e-05 2023-11-01T04:16:53.165 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.165 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.165 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.165 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 69/31 ECSubWrite(tid=84, reqid=client.4559.0:563, at_version=69'66, trim_to=0'0, roll_forward_to=68'62))", 2023-11-01T04:16:53.166 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:49.712801+0000", 2023-11-01T04:16:53.166 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.239449741, 2023-11-01T04:16:53.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104872064, 2023-11-01T04:16:53.166 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.166 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.166 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.712801+0000", 2023-11-01T04:16:53.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.712801+0000", 2023-11-01T04:16:53.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.712803+0000", 2023-11-01T04:16:53.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.384e-06 2023-11-01T04:16:53.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.712831+0000", 2023-11-01T04:16:53.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8109000000000001e-05 2023-11-01T04:16:53.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.712853+0000", 2023-11-01T04:16:53.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2096e-05 2023-11-01T04:16:53.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.712871+0000", 2023-11-01T04:16:53.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8050000000000002e-05 2023-11-01T04:16:53.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.815289+0000", 2023-11-01T04:16:53.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102418603 2023-11-01T04:16:53.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.815307+0000", 2023-11-01T04:16:53.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7246000000000001e-05 2023-11-01T04:16:53.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.817623+0000", 2023-11-01T04:16:53.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002316317 2023-11-01T04:16:53.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.817673+0000", 2023-11-01T04:16:53.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0259000000000001e-05 2023-11-01T04:16:53.175 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.175 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.176 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.176 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.176 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.176 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:53.177 INFO:tasks.rados.rados.0.smithi190.stdout:248: finishing write tid 1 to smithi19039307-44 2023-11-01T04:16:53.177 INFO:tasks.rados.rados.0.smithi190.stdout:248: finishing write tid 2 to smithi19039307-44 2023-11-01T04:16:53.178 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:53.178 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-01T04:16:53.179 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:53.179 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-01T04:16:53.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-01T04:16:53.180 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:53.181 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-01T04:16:53.181 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:53.181 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-01T04:16:53.182 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:53.182 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-01T04:16:53.182 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:53.174+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:53.229 INFO:tasks.rados.rados.0.smithi190.stdout:248: finishing write tid 3 to smithi19039307-44 2023-11-01T04:16:53.278 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:53.275+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:53.330 INFO:tasks.rados.rados.0.smithi190.stdout:248: finishing write tid 4 to smithi19039307-44 2023-11-01T04:16:53.343 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:53.344 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-11-01T04:16:53.351 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:53.352 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:53.352 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2023-11-01T04:16:53.352 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:53.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-01T04:16:53.379 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:53.379 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-01T04:16:53.398 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:53.398 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-01T04:16:53.413 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:53.413+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:53.428 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:53.428 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-01T04:16:53.454 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:53.454 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-01T04:16:53.455 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.12 2023-11-01T04:16:53.479 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:53.475+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:53.480 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:53.480 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-01T04:16:53.484 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:53.484 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:53.484 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:16:53.484 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:16:53.484 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:53.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-01T04:16:53.563 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:53.563 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-01T04:16:53.580 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:53.580+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:53.597 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:53.598 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-01T04:16:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-01T04:16:53.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-01T04:16:53.667 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:53.668 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:16:53.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:16:53.668 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:16:53.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.669 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:166 3.8s0 3:1b945fb3:::smithi19039307-3:head [read 0~49152,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:16:53.669 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.317340+0000", 2023-11-01T04:16:53.669 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 37.347927554999998, 2023-11-01T04:16:53.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.405887145, 2023-11-01T04:16:53.669 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.670 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:53.670 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:53.670 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:53.670 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:53.670 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 166 2023-11-01T04:16:53.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.671 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317340+0000", 2023-11-01T04:16:53.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317340+0000", 2023-11-01T04:16:53.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317341+0000", 2023-11-01T04:16:53.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.077e-06 2023-11-01T04:16:53.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317350+0000", 2023-11-01T04:16:53.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0329999999999997e-06 2023-11-01T04:16:53.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317367+0000", 2023-11-01T04:16:53.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6612999999999998e-05 2023-11-01T04:16:53.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317380+0000", 2023-11-01T04:16:53.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2832e-05 2023-11-01T04:16:53.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.421004+0000", 2023-11-01T04:16:53.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.103624181 2023-11-01T04:16:53.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:53.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.421215+0000", 2023-11-01T04:16:53.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00021057199999999999 2023-11-01T04:16:53.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.723227+0000", 2023-11-01T04:16:53.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30201283699999998 2023-11-01T04:16:53.679 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.679 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.680 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.680 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:326 3.8s0 3:1514c4b8:::smithi19039307-15:head [append 0~376832] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:53.680 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.056621+0000", 2023-11-01T04:16:53.681 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.608646652000001, 2023-11-01T04:16:53.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43382757900000002, 2023-11-01T04:16:53.681 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.681 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:53.681 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:53.682 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:53.682 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:53.682 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 326 2023-11-01T04:16:53.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.682 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.056621+0000", 2023-11-01T04:16:53.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.056621+0000", 2023-11-01T04:16:53.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.056623+0000", 2023-11-01T04:16:53.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2640000000000001e-06 2023-11-01T04:16:53.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057077+0000", 2023-11-01T04:16:53.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00045417599999999998 2023-11-01T04:16:53.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057092+0000", 2023-11-01T04:16:53.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5138e-05 2023-11-01T04:16:53.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057103+0000", 2023-11-01T04:16:53.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1117999999999999e-05 2023-11-01T04:16:53.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.159853+0000", 2023-11-01T04:16:53.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10274968399999999 2023-11-01T04:16:53.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:53.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.159948+0000", 2023-11-01T04:16:53.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5097000000000001e-05 2023-11-01T04:16:53.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.160626+0000", 2023-11-01T04:16:53.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00067830199999999998 2023-11-01T04:16:53.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162313+0000", 2023-11-01T04:16:53.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016867799999999999 2023-11-01T04:16:53.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:53.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.490341+0000", 2023-11-01T04:16:53.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32802831100000002 2023-11-01T04:16:53.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.490449+0000", 2023-11-01T04:16:53.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010770899999999999 2023-11-01T04:16:53.694 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.694 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.694 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.694 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:327 3.8s0 3:1514c4b8:::smithi19039307-15:head [setxattr _header (56) in=63b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:53.695 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.057121+0000", 2023-11-01T04:16:53.695 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.608146592000001, 2023-11-01T04:16:53.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.63439970300000004, 2023-11-01T04:16:53.695 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.695 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:53.696 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:53.696 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:53.696 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:53.696 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 327 2023-11-01T04:16:53.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.696 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057121+0000", 2023-11-01T04:16:53.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057121+0000", 2023-11-01T04:16:53.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057122+0000", 2023-11-01T04:16:53.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8199999999999999e-07 2023-11-01T04:16:53.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057126+0000", 2023-11-01T04:16:53.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0439999999999997e-06 2023-11-01T04:16:53.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057134+0000", 2023-11-01T04:16:53.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.456e-06 2023-11-01T04:16:53.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057141+0000", 2023-11-01T04:16:53.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.861e-06 2023-11-01T04:16:53.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.161271+0000", 2023-11-01T04:16:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104129201 2023-11-01T04:16:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:53.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.161389+0000", 2023-11-01T04:16:53.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011806 2023-11-01T04:16:53.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.161906+0000", 2023-11-01T04:16:53.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00051756700000000003 2023-11-01T04:16:53.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162616+0000", 2023-11-01T04:16:53.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00070943799999999995 2023-11-01T04:16:53.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:53.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691275+0000", 2023-11-01T04:16:53.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52865958499999999 2023-11-01T04:16:53.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691521+0000", 2023-11-01T04:16:53.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024580899999999999 2023-11-01T04:16:53.708 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.708 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.708 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.709 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:328 3.8s0 3:1514c4b8:::smithi19039307-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:53.709 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.057152+0000", 2023-11-01T04:16:53.709 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.608116044999999, 2023-11-01T04:16:53.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83879504599999999, 2023-11-01T04:16:53.709 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.710 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:53.710 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:53.710 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:53.710 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:53.710 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 328 2023-11-01T04:16:53.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.711 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057152+0000", 2023-11-01T04:16:53.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057152+0000", 2023-11-01T04:16:53.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057152+0000", 2023-11-01T04:16:53.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5599999999999995e-07 2023-11-01T04:16:53.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057155+0000", 2023-11-01T04:16:53.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5050000000000002e-06 2023-11-01T04:16:53.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057163+0000", 2023-11-01T04:16:53.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1629999999999996e-06 2023-11-01T04:16:53.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057169+0000", 2023-11-01T04:16:53.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8930000000000002e-06 2023-11-01T04:16:53.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162181+0000", 2023-11-01T04:16:53.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105011939 2023-11-01T04:16:53.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2023-11-01T04:16:53.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162296+0000", 2023-11-01T04:16:53.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115392 2023-11-01T04:16:53.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691675+0000", 2023-11-01T04:16:53.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52937891999999998 2023-11-01T04:16:53.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:53.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691837+0000", 2023-11-01T04:16:53.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016180099999999999 2023-11-01T04:16:53.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.895947+0000", 2023-11-01T04:16:53.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.204109877 2023-11-01T04:16:53.720 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.721 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.721 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.721 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:339 3.8s0 3:19d11dee:::smithi19039307-39:head [append 0~737280 in=737280b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:53.722 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.185406+0000", 2023-11-01T04:16:53.722 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.479862118, 2023-11-01T04:16:53.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70798112199999996, 2023-11-01T04:16:53.722 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.722 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:53.722 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:53.723 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:53.723 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:53.723 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 339 2023-11-01T04:16:53.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.723 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.185406+0000", 2023-11-01T04:16:53.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.185406+0000", 2023-11-01T04:16:53.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.185408+0000", 2023-11-01T04:16:53.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9530000000000002e-06 2023-11-01T04:16:53.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186621+0000", 2023-11-01T04:16:53.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012132270000000001 2023-11-01T04:16:53.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186644+0000", 2023-11-01T04:16:53.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2878e-05 2023-11-01T04:16:53.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186663+0000", 2023-11-01T04:16:53.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9650999999999999e-05 2023-11-01T04:16:53.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.729 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186754+0000", 2023-11-01T04:16:53.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0606999999999995e-05 2023-11-01T04:16:53.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.730 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:53.730 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186902+0000", 2023-11-01T04:16:53.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000147448 2023-11-01T04:16:53.730 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.731 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.188122+0000", 2023-11-01T04:16:53.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001220732 2023-11-01T04:16:53.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.191051+0000", 2023-11-01T04:16:53.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029283669999999999 2023-11-01T04:16:53.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:53.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893331+0000", 2023-11-01T04:16:53.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70228026200000004 2023-11-01T04:16:53.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893387+0000", 2023-11-01T04:16:53.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5996999999999999e-05 2023-11-01T04:16:53.734 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.734 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.734 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.735 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:340 3.8s0 3:19d11dee:::smithi19039307-39:head [append 0~376832 in=376832b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:53.735 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.186694+0000", 2023-11-01T04:16:53.735 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.478574268999999, 2023-11-01T04:16:53.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70680890500000004, 2023-11-01T04:16:53.735 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.736 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:53.736 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:53.736 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:53.736 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:53.736 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 340 2023-11-01T04:16:53.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.737 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186694+0000", 2023-11-01T04:16:53.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186694+0000", 2023-11-01T04:16:53.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186695+0000", 2023-11-01T04:16:53.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3400000000000001e-06 2023-11-01T04:16:53.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187348+0000", 2023-11-01T04:16:53.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00065316300000000001 2023-11-01T04:16:53.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187364+0000", 2023-11-01T04:16:53.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5979999999999999e-05 2023-11-01T04:16:53.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187372+0000", 2023-11-01T04:16:53.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6599999999999995e-06 2023-11-01T04:16:53.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.189420+0000", 2023-11-01T04:16:53.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020482959999999998 2023-11-01T04:16:53.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:53.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.189500+0000", 2023-11-01T04:16:53.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9621000000000002e-05 2023-11-01T04:16:53.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.744 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.190386+0000", 2023-11-01T04:16:53.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00088641100000000003 2023-11-01T04:16:53.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.191553+0000", 2023-11-01T04:16:53.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001166413 2023-11-01T04:16:53.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:53.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893466+0000", 2023-11-01T04:16:53.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.701913911 2023-11-01T04:16:53.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893503+0000", 2023-11-01T04:16:53.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6109999999999998e-05 2023-11-01T04:16:53.747 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.747 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.747 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.748 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:341 3.8s0 3:19d11dee:::smithi19039307-39:head [setxattr _header (56) in=63b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:53.748 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.187391+0000", 2023-11-01T04:16:53.748 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.477876679000001, 2023-11-01T04:16:53.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70629933600000006, 2023-11-01T04:16:53.749 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.749 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:53.749 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:53.749 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:53.750 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:53.750 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 341 2023-11-01T04:16:53.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.750 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187391+0000", 2023-11-01T04:16:53.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.752 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187391+0000", 2023-11-01T04:16:53.752 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.752 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.752 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.752 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.752 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187392+0000", 2023-11-01T04:16:53.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7100000000000011e-07 2023-11-01T04:16:53.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.753 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187398+0000", 2023-11-01T04:16:53.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3329999999999996e-06 2023-11-01T04:16:53.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.754 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.754 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187407+0000", 2023-11-01T04:16:53.754 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6269999999999995e-06 2023-11-01T04:16:53.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187419+0000", 2023-11-01T04:16:53.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2336e-05 2023-11-01T04:16:53.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.756 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.756 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.265764+0000", 2023-11-01T04:16:53.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.078345021000000001 2023-11-01T04:16:53.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:53.757 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.265926+0000", 2023-11-01T04:16:53.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016117900000000001 2023-11-01T04:16:53.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.758 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.266578+0000", 2023-11-01T04:16:53.758 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00065216799999999995 2023-11-01T04:16:53.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.758 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.758 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.758 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.267885+0000", 2023-11-01T04:16:53.759 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013072699999999999 2023-11-01T04:16:53.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.759 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.759 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:53.759 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893617+0000", 2023-11-01T04:16:53.760 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62573200299999998 2023-11-01T04:16:53.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.760 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.760 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.760 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893691+0000", 2023-11-01T04:16:53.760 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3428000000000002e-05 2023-11-01T04:16:53.761 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.761 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.761 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.761 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.762 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:342 3.8s0 3:19d11dee:::smithi19039307-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:53.762 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.187433+0000", 2023-11-01T04:16:53.762 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.477834753, 2023-11-01T04:16:53.762 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.009816936, 2023-11-01T04:16:53.762 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.762 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:53.763 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:53.763 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:53.763 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:53.763 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 342 2023-11-01T04:16:53.763 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.763 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.764 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.764 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.764 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187433+0000", 2023-11-01T04:16:53.764 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.764 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.764 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.765 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.765 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187433+0000", 2023-11-01T04:16:53.765 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.765 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.765 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.766 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.766 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187434+0000", 2023-11-01T04:16:53.766 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.75e-07 2023-11-01T04:16:53.766 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.766 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.766 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.767 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187437+0000", 2023-11-01T04:16:53.767 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1870000000000001e-06 2023-11-01T04:16:53.767 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.767 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.767 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.767 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187446+0000", 2023-11-01T04:16:53.768 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0499999999999997e-06 2023-11-01T04:16:53.768 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.768 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.768 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.768 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.287537+0000", 2023-11-01T04:16:53.769 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10009085 2023-11-01T04:16:53.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.769 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.769 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.769 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.288584+0000", 2023-11-01T04:16:53.769 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001047463 2023-11-01T04:16:53.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.770 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.770 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2023-11-01T04:16:53.770 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.288883+0000", 2023-11-01T04:16:53.770 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029857700000000001 2023-11-01T04:16:53.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.771 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.771 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.771 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.894227+0000", 2023-11-01T04:16:53.771 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60534433899999995 2023-11-01T04:16:53.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.772 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.772 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:53.772 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.894404+0000", 2023-11-01T04:16:53.772 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000176732 2023-11-01T04:16:53.772 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.772 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.773 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.773 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.197250+0000", 2023-11-01T04:16:53.773 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.302846163 2023-11-01T04:16:53.773 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.773 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.774 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.774 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.774 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=28, last_complete=46'20, committed=1, applied=1))", 2023-11-01T04:16:53.774 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.390586+0000", 2023-11-01T04:16:53.774 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.274681867000002, 2023-11-01T04:16:53.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30212400700000003, 2023-11-01T04:16:53.775 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.775 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.775 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.775 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.775 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.776 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390586+0000", 2023-11-01T04:16:53.776 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.776 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.776 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.777 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390586+0000", 2023-11-01T04:16:53.777 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.777 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.777 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.777 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.777 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390587+0000", 2023-11-01T04:16:53.778 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9999999999999997e-07 2023-11-01T04:16:53.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.778 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.778 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.778 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390592+0000", 2023-11-01T04:16:53.778 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9400000000000001e-06 2023-11-01T04:16:53.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.779 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.779 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.779 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390607+0000", 2023-11-01T04:16:53.779 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5216e-05 2023-11-01T04:16:53.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.780 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.780 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.780 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390617+0000", 2023-11-01T04:16:53.780 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8260000000000005e-06 2023-11-01T04:16:53.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.781 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.781 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.781 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.692681+0000", 2023-11-01T04:16:53.781 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30206392300000001 2023-11-01T04:16:53.781 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.781 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.782 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.782 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.692710+0000", 2023-11-01T04:16:53.782 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9502e-05 2023-11-01T04:16:53.782 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.782 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.782 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.783 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.783 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.783 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=29, last_complete=46'21, committed=1, applied=1))", 2023-11-01T04:16:53.783 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.391131+0000", 2023-11-01T04:16:53.783 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.274136730999999, 2023-11-01T04:16:53.783 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30166817800000001, 2023-11-01T04:16:53.784 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.784 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.784 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.784 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.784 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.784 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391131+0000", 2023-11-01T04:16:53.785 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.785 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.785 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.785 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.785 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391131+0000", 2023-11-01T04:16:53.785 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.786 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.786 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.786 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.786 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391132+0000", 2023-11-01T04:16:53.786 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5000000000000001e-07 2023-11-01T04:16:53.787 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.787 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.787 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.787 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391138+0000", 2023-11-01T04:16:53.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1999999999999999e-06 2023-11-01T04:16:53.787 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.788 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391156+0000", 2023-11-01T04:16:53.788 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7252999999999999e-05 2023-11-01T04:16:53.788 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.789 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391165+0000", 2023-11-01T04:16:53.789 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3530000000000001e-06 2023-11-01T04:16:53.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.789 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.790 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.692767+0000", 2023-11-01T04:16:53.790 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30160192600000002 2023-11-01T04:16:53.790 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.790 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.790 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.790 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.692799+0000", 2023-11-01T04:16:53.791 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2496000000000002e-05 2023-11-01T04:16:53.791 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.791 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.791 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.792 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:16:53.792 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:16:53.792 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.272929323, 2023-11-01T04:16:53.792 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40054957200000002, 2023-11-01T04:16:53.792 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.793 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.793 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.793 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.793 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.793 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:16:53.793 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.794 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.794 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.794 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:16:53.794 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.795 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.795 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.795 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:16:53.795 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7400000000000002e-07 2023-11-01T04:16:53.795 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.795 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.796 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.796 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392345+0000", 2023-11-01T04:16:53.796 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8690000000000001e-06 2023-11-01T04:16:53.796 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.796 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.797 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.797 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392362+0000", 2023-11-01T04:16:53.797 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6194000000000001e-05 2023-11-01T04:16:53.797 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.797 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.797 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.798 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392372+0000", 2023-11-01T04:16:53.798 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0692e-05 2023-11-01T04:16:53.798 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.798 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.798 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.798 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792857+0000", 2023-11-01T04:16:53.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40048482800000001 2023-11-01T04:16:53.799 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.799 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.799 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.799 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792888+0000", 2023-11-01T04:16:53.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1115000000000003e-05 2023-11-01T04:16:53.800 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.800 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.800 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.800 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.800 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.800 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=28, last_complete=46'20, committed=1, applied=1))", 2023-11-01T04:16:53.801 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.393704+0000", 2023-11-01T04:16:53.801 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.271563806, 2023-11-01T04:16:53.801 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39925663099999997, 2023-11-01T04:16:53.801 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.801 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.802 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.802 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.802 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393704+0000", 2023-11-01T04:16:53.802 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393704+0000", 2023-11-01T04:16:53.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.803 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.804 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.804 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393705+0000", 2023-11-01T04:16:53.804 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3699999999999999e-07 2023-11-01T04:16:53.804 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.804 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.804 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.805 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393711+0000", 2023-11-01T04:16:53.805 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6899999999999997e-06 2023-11-01T04:16:53.805 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.805 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.805 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.806 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393726+0000", 2023-11-01T04:16:53.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5559000000000001e-05 2023-11-01T04:16:53.806 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.806 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.806 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.806 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393735+0000", 2023-11-01T04:16:53.807 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2830000000000001e-06 2023-11-01T04:16:53.807 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.807 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.807 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.807 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792931+0000", 2023-11-01T04:16:53.807 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39919643599999999 2023-11-01T04:16:53.808 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.808 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.808 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.808 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792961+0000", 2023-11-01T04:16:53.808 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9726e-05 2023-11-01T04:16:53.809 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.809 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.809 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.809 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.809 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.809 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=29, last_complete=46'21, committed=1, applied=1))", 2023-11-01T04:16:53.810 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:16:53.810 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.271504927999999, 2023-11-01T04:16:53.810 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39925449800000001, 2023-11-01T04:16:53.810 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.810 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.810 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.811 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.811 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.811 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:16:53.811 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.811 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.812 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.812 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.812 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:16:53.812 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.812 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.813 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:16:53.813 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7700000000000005e-07 2023-11-01T04:16:53.813 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.814 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.814 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393770+0000", 2023-11-01T04:16:53.814 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.28e-06 2023-11-01T04:16:53.814 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.814 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.814 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.815 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393779+0000", 2023-11-01T04:16:53.815 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6760000000000002e-06 2023-11-01T04:16:53.815 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.815 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.815 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.815 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393786+0000", 2023-11-01T04:16:53.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8229999999999997e-06 2023-11-01T04:16:53.816 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.816 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.816 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.816 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792996+0000", 2023-11-01T04:16:53.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39921008299999999 2023-11-01T04:16:53.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.817 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.793017+0000", 2023-11-01T04:16:53.817 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1158999999999999e-05 2023-11-01T04:16:53.817 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.818 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.818 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.818 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.818 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:16:53.819 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.394729+0000", 2023-11-01T04:16:53.819 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.270539022000001, 2023-11-01T04:16:53.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.49843099299999999, 2023-11-01T04:16:53.819 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.819 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.819 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.820 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.820 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394729+0000", 2023-11-01T04:16:53.820 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.820 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.821 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.821 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394729+0000", 2023-11-01T04:16:53.821 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.821 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.821 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.822 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.822 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394730+0000", 2023-11-01T04:16:53.822 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.093e-06 2023-11-01T04:16:53.822 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.822 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.822 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394738+0000", 2023-11-01T04:16:53.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2300000000000008e-06 2023-11-01T04:16:53.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.823 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394760+0000", 2023-11-01T04:16:53.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1679999999999999e-05 2023-11-01T04:16:53.824 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.824 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.824 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.824 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.395298+0000", 2023-11-01T04:16:53.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053837199999999996 2023-11-01T04:16:53.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.825 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.825 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.825 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893130+0000", 2023-11-01T04:16:53.825 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.49783206600000002 2023-11-01T04:16:53.826 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.826 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.826 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.826 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893160+0000", 2023-11-01T04:16:53.826 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9552000000000001e-05 2023-11-01T04:16:53.826 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.827 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.827 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.827 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.827 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.827 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:16:53.827 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.468843+0000", 2023-11-01T04:16:53.828 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.196425172000001, 2023-11-01T04:16:53.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42441382100000002, 2023-11-01T04:16:53.828 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.828 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.828 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.829 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.829 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468843+0000", 2023-11-01T04:16:53.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.829 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.830 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468843+0000", 2023-11-01T04:16:53.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.830 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.830 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.830 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.831 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468845+0000", 2023-11-01T04:16:53.831 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.21e-06 2023-11-01T04:16:53.831 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.831 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.832 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468857+0000", 2023-11-01T04:16:53.832 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1678999999999999e-05 2023-11-01T04:16:53.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.832 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.832 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.832 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468888+0000", 2023-11-01T04:16:53.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1788999999999997e-05 2023-11-01T04:16:53.833 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.833 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.833 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569025+0000", 2023-11-01T04:16:53.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100136424 2023-11-01T04:16:53.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893225+0000", 2023-11-01T04:16:53.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32420036899999999 2023-11-01T04:16:53.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893257+0000", 2023-11-01T04:16:53.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1350000000000003e-05 2023-11-01T04:16:53.835 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.836 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.836 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.836 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.836 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.836 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=28, last_complete=46'20, committed=1, applied=1))", 2023-11-01T04:16:53.836 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.569161+0000", 2023-11-01T04:16:53.837 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.096106893999998, 2023-11-01T04:16:53.837 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32424099299999998, 2023-11-01T04:16:53.837 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.837 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.837 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569161+0000", 2023-11-01T04:16:53.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.839 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.839 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569161+0000", 2023-11-01T04:16:53.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.839 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.839 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569162+0000", 2023-11-01T04:16:53.840 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3400000000000001e-06 2023-11-01T04:16:53.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.840 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.841 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569173+0000", 2023-11-01T04:16:53.841 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0293e-05 2023-11-01T04:16:53.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.841 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.841 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.841 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569197+0000", 2023-11-01T04:16:53.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4156e-05 2023-11-01T04:16:53.842 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.842 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.842 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.842 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569213+0000", 2023-11-01T04:16:53.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6455000000000001e-05 2023-11-01T04:16:53.843 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.843 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.843 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.843 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893271+0000", 2023-11-01T04:16:53.843 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32405778099999999 2023-11-01T04:16:53.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.844 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.844 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.844 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893402+0000", 2023-11-01T04:16:53.844 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000130968 2023-11-01T04:16:53.844 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=29, last_complete=46'21, committed=1, applied=1))", 2023-11-01T04:16:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.569263+0000", 2023-11-01T04:16:53.846 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.096004413999999, 2023-11-01T04:16:53.846 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32424595, 2023-11-01T04:16:53.846 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.846 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.846 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.847 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.847 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.847 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569263+0000", 2023-11-01T04:16:53.847 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.847 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.847 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.848 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.848 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569263+0000", 2023-11-01T04:16:53.848 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.848 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.848 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.848 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.849 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569264+0000", 2023-11-01T04:16:53.849 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7700000000000004e-07 2023-11-01T04:16:53.849 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.849 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.849 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.850 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569269+0000", 2023-11-01T04:16:53.850 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9490000000000004e-06 2023-11-01T04:16:53.850 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.850 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.850 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.850 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569282+0000", 2023-11-01T04:16:53.851 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3169999999999999e-05 2023-11-01T04:16:53.851 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.851 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.851 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.851 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569293+0000", 2023-11-01T04:16:53.852 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0526000000000001e-05 2023-11-01T04:16:53.852 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.852 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.852 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.852 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893418+0000", 2023-11-01T04:16:53.852 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32412480999999999 2023-11-01T04:16:53.853 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.853 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.853 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.853 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893509+0000", 2023-11-01T04:16:53.853 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1817999999999994e-05 2023-11-01T04:16:53.853 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.854 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.854 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.854 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.854 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.854 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:16:53.855 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.569435+0000", 2023-11-01T04:16:53.855 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.095832439999999, 2023-11-01T04:16:53.855 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32468964500000003, 2023-11-01T04:16:53.855 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.855 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.855 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.856 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.856 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.856 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569435+0000", 2023-11-01T04:16:53.856 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.856 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.856 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.857 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.857 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569435+0000", 2023-11-01T04:16:53.857 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.857 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.857 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.857 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.858 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569436+0000", 2023-11-01T04:16:53.858 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6700000000000003e-07 2023-11-01T04:16:53.858 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.858 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.858 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.858 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569441+0000", 2023-11-01T04:16:53.859 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3059999999999997e-06 2023-11-01T04:16:53.859 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.859 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.859 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.859 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569461+0000", 2023-11-01T04:16:53.860 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9856000000000001e-05 2023-11-01T04:16:53.860 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.860 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.860 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.860 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.669600+0000", 2023-11-01T04:16:53.860 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100139114 2023-11-01T04:16:53.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.861 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.861 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.861 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893551+0000", 2023-11-01T04:16:53.861 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.223950963 2023-11-01T04:16:53.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.862 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.862 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.862 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.894125+0000", 2023-11-01T04:16:53.862 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00057373899999999998 2023-11-01T04:16:53.862 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.862 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.863 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.863 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.863 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.863 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(osd.12.22:5 3.8s0 3:19d11dee:::smithi19039307-39:head [assert-version v22,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:53.863 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.454207+0000", 2023-11-01T04:16:53.864 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 26.211060491000001, 2023-11-01T04:16:53.864 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35587170800000001, 2023-11-01T04:16:53.864 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.864 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:53.864 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.864 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.865 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.865 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454207+0000", 2023-11-01T04:16:53.865 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.865 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.865 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.865 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.866 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454207+0000", 2023-11-01T04:16:53.866 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.866 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.866 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.866 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.867 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454210+0000", 2023-11-01T04:16:53.867 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2299999999999998e-06 2023-11-01T04:16:53.867 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.867 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.867 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.867 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454219+0000", 2023-11-01T04:16:53.868 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0469999999999999e-06 2023-11-01T04:16:53.868 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.868 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.868 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.868 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454239+0000", 2023-11-01T04:16:53.868 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0499000000000001e-05 2023-11-01T04:16:53.869 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.869 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.869 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.869 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454256+0000", 2023-11-01T04:16:53.869 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6993000000000001e-05 2023-11-01T04:16:53.869 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.870 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.870 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.870 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454343+0000", 2023-11-01T04:16:53.870 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.687e-05 2023-11-01T04:16:53.870 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.871 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:53.871 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454435+0000", 2023-11-01T04:16:53.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2019000000000001e-05 2023-11-01T04:16:53.871 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.872 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.810079+0000", 2023-11-01T04:16:53.872 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35564404999999999 2023-11-01T04:16:53.872 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.872 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.872 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.873 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.873 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.873 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.0s4 68/31 ECSubWrite(tid=36, reqid=osd.3.0:36, at_version=68'21, trim_to=0'0, roll_forward_to=67'19))", 2023-11-01T04:16:53.873 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.894669+0000", 2023-11-01T04:16:53.873 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.7705984109999999, 2023-11-01T04:16:53.874 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31005245300000001, 2023-11-01T04:16:53.874 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:53.874 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:53.874 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:53.874 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.874 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:53.875 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894669+0000", 2023-11-01T04:16:53.875 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.875 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.875 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.875 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:53.875 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894669+0000", 2023-11-01T04:16:53.876 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:53.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.876 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:53.876 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894670+0000", 2023-11-01T04:16:53.876 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9399999999999995e-07 2023-11-01T04:16:53.877 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.877 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.877 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:53.877 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894722+0000", 2023-11-01T04:16:53.877 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1962000000000002e-05 2023-11-01T04:16:53.878 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.878 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.878 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:53.878 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894772+0000", 2023-11-01T04:16:53.878 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9771999999999997e-05 2023-11-01T04:16:53.878 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.879 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.879 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:53.879 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.994910+0000", 2023-11-01T04:16:53.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100138189 2023-11-01T04:16:53.879 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.879 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.880 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:53.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.104098+0000", 2023-11-01T04:16:53.880 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10918811 2023-11-01T04:16:53.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.881 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:53.881 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.104110+0000", 2023-11-01T04:16:53.881 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1787000000000001e-05 2023-11-01T04:16:53.881 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.881 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.882 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:53.882 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.204645+0000", 2023-11-01T04:16:53.882 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100534978 2023-11-01T04:16:53.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:53.882 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:53.883 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:53.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.204722+0000", 2023-11-01T04:16:53.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7161000000000002e-05 2023-11-01T04:16:53.883 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.883 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.883 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.884 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:53.884 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:53.884 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:53.885 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-11-01T04:16:53.886 INFO:tasks.rados.rados.0.smithi190.stdout:248: finishing write tid 5 to smithi19039307-44 2023-11-01T04:16:53.886 INFO:tasks.rados.rados.0.smithi190.stdout:248: oid 44 updating version 0 to 14 2023-11-01T04:16:53.886 INFO:tasks.rados.rados.0.smithi190.stdout:248: oid 44 version 14 is already newer than 13 2023-11-01T04:16:53.886 INFO:tasks.rados.rados.0.smithi190.stdout:248: oid 44 version 14 is already newer than 12 2023-11-01T04:16:53.887 INFO:tasks.rados.rados.0.smithi190.stdout:248: oid 44 version 14 is already newer than 11 2023-11-01T04:16:53.887 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 44 v 14 (ObjNum 102 snap 21 seq_num 102) dirty exists 2023-11-01T04:16:53.887 INFO:tasks.rados.rados.0.smithi190.stdout:248: left oid 44 (ObjNum 102 snap 21 seq_num 102) 2023-11-01T04:16:53.887 INFO:tasks.rados.rados.0.smithi190.stdout:242: done (4 left) 2023-11-01T04:16:53.887 INFO:tasks.rados.rados.0.smithi190.stdout:248: done (3 left) 2023-11-01T04:16:53.888 INFO:tasks.rados.rados.0.smithi190.stdout:251: done (2 left) 2023-11-01T04:16:53.888 INFO:tasks.rados.rados.0.smithi190.stdout:252: snap_create 2023-11-01T04:16:53.888 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:53.882+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:53.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-01T04:16:53.986 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:53.985+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:54.025 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:54.025 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:54.025 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2023-11-01T04:16:54.025 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:54.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-01T04:16:54.129 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:54.129+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:54.188 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:54.187+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:54.205 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:54.205 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:54.205 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:16:54.205 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:16:54.206 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:54.219 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-01T04:16:54.239 INFO:tasks.rados.rados.0.smithi190.stdout:249: finishing write tid 1 to smithi19039307-5 2023-11-01T04:16:54.242 INFO:tasks.rados.rados.0.smithi190.stdout:249: finishing write tid 2 to smithi19039307-5 2023-11-01T04:16:54.289 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:54.288+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:54.343 INFO:tasks.rados.rados.0.smithi190.stdout:249: finishing write tid 3 to smithi19039307-5 2023-11-01T04:16:54.386 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:54.387 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:16:54.387 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:16:54.387 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:16:54.387 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.387 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.6s6 36/31 ECSubWrite(tid=22, reqid=client.4559.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=36'4))", 2023-11-01T04:16:54.388 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.688060+0000", 2023-11-01T04:16:54.388 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.697924473, 2023-11-01T04:16:54.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19558595100000001, 2023-11-01T04:16:54.388 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.388 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.389 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.389 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.389 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.389 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688060+0000", 2023-11-01T04:16:54.389 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.390 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.390 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.390 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688060+0000", 2023-11-01T04:16:54.390 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.391 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.391 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.391 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.391 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688062+0000", 2023-11-01T04:16:54.391 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1859999999999999e-06 2023-11-01T04:16:54.392 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.392 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.392 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.392 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688084+0000", 2023-11-01T04:16:54.392 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1849e-05 2023-11-01T04:16:54.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.393 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.393 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.393 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688115+0000", 2023-11-01T04:16:54.393 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0788999999999999e-05 2023-11-01T04:16:54.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.394 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.394 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.394 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688131+0000", 2023-11-01T04:16:54.394 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6506000000000001e-05 2023-11-01T04:16:54.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.395 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.395 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.395 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.788338+0000", 2023-11-01T04:16:54.395 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100206936 2023-11-01T04:16:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.788353+0000", 2023-11-01T04:16:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5109000000000001e-05 2023-11-01T04:16:54.396 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.883595+0000", 2023-11-01T04:16:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.095241261999999993 2023-11-01T04:16:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.883646+0000", 2023-11-01T04:16:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1313999999999997e-05 2023-11-01T04:16:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.398 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 37/31 ECSubWrite(tid=20, reqid=client.4559.0:185, at_version=37'13, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:16:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.965368+0000", 2023-11-01T04:16:54.399 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 37.420616684000002, 2023-11-01T04:16:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10412571, 2023-11-01T04:16:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965368+0000", 2023-11-01T04:16:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965368+0000", 2023-11-01T04:16:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965369+0000", 2023-11-01T04:16:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4600000000000004e-07 2023-11-01T04:16:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.403 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965431+0000", 2023-11-01T04:16:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2322000000000004e-05 2023-11-01T04:16:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.404 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965465+0000", 2023-11-01T04:16:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4408e-05 2023-11-01T04:16:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.405 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965490+0000", 2023-11-01T04:16:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4272000000000001e-05 2023-11-01T04:16:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965716+0000", 2023-11-01T04:16:54.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022658800000000001 2023-11-01T04:16:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965742+0000", 2023-11-01T04:16:54.407 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5445000000000001e-05 2023-11-01T04:16:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.069410+0000", 2023-11-01T04:16:54.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10366858299999999 2023-11-01T04:16:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.069494+0000", 2023-11-01T04:16:54.409 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3446e-05 2023-11-01T04:16:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 37/31 ECSubWrite(tid=21, reqid=client.4559.0:186, at_version=37'14, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:16:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.967795+0000", 2023-11-01T04:16:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 37.418189407, 2023-11-01T04:16:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102571579, 2023-11-01T04:16:54.411 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967795+0000", 2023-11-01T04:16:54.412 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967795+0000", 2023-11-01T04:16:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967797+0000", 2023-11-01T04:16:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5600000000000001e-06 2023-11-01T04:16:54.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967838+0000", 2023-11-01T04:16:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1578000000000001e-05 2023-11-01T04:16:54.415 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967874+0000", 2023-11-01T04:16:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5846000000000002e-05 2023-11-01T04:16:54.416 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.967898+0000", 2023-11-01T04:16:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3402000000000001e-05 2023-11-01T04:16:54.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.068196+0000", 2023-11-01T04:16:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100298416 2023-11-01T04:16:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.068223+0000", 2023-11-01T04:16:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7248000000000001e-05 2023-11-01T04:16:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.070268+0000", 2023-11-01T04:16:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020447289999999999 2023-11-01T04:16:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.070367+0000", 2023-11-01T04:16:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8800000000000003e-05 2023-11-01T04:16:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.421 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 37/31 ECSubWrite(tid=24, reqid=client.4559.0:194, at_version=37'17, trim_to=0'0, roll_forward_to=37'15))", 2023-11-01T04:16:54.422 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.479841+0000", 2023-11-01T04:16:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 36.906143679000003, 2023-11-01T04:16:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102648423, 2023-11-01T04:16:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.423 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.479841+0000", 2023-11-01T04:16:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.479841+0000", 2023-11-01T04:16:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.479843+0000", 2023-11-01T04:16:54.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2069999999999998e-06 2023-11-01T04:16:54.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.480582+0000", 2023-11-01T04:16:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00073883 2023-11-01T04:16:54.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.480614+0000", 2023-11-01T04:16:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2149e-05 2023-11-01T04:16:54.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.480633+0000", 2023-11-01T04:16:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9315999999999999e-05 2023-11-01T04:16:54.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.580829+0000", 2023-11-01T04:16:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10019570799999999 2023-11-01T04:16:54.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.580837+0000", 2023-11-01T04:16:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9209999999999992e-06 2023-11-01T04:16:54.431 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.582452+0000", 2023-11-01T04:16:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001615099 2023-11-01T04:16:54.432 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.582489+0000", 2023-11-01T04:16:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7193000000000001e-05 2023-11-01T04:16:54.433 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.434 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 41/31 ECSubWrite(tid=38, reqid=osd.11.0:38, at_version=41'29, trim_to=0'0, roll_forward_to=41'27))", 2023-11-01T04:16:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:21.646162+0000", 2023-11-01T04:16:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 32.739822197999999, 2023-11-01T04:16:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102766129, 2023-11-01T04:16:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.435 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646162+0000", 2023-11-01T04:16:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.436 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646162+0000", 2023-11-01T04:16:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646164+0000", 2023-11-01T04:16:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.103e-06 2023-11-01T04:16:54.438 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646219+0000", 2023-11-01T04:16:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4981000000000001e-05 2023-11-01T04:16:54.439 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646272+0000", 2023-11-01T04:16:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2324999999999998e-05 2023-11-01T04:16:54.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.646298+0000", 2023-11-01T04:16:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6622e-05 2023-11-01T04:16:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.746561+0000", 2023-11-01T04:16:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100262254 2023-11-01T04:16:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.746586+0000", 2023-11-01T04:16:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5015e-05 2023-11-01T04:16:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.443 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.748844+0000", 2023-11-01T04:16:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022586849999999999 2023-11-01T04:16:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.444 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:21.748928+0000", 2023-11-01T04:16:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4143999999999999e-05 2023-11-01T04:16:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.445 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.7s2 43/31 ECSubWrite(tid=37, reqid=osd.14.0:37, at_version=43'16, trim_to=0'0, roll_forward_to=41'14))", 2023-11-01T04:16:54.446 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:23.017682+0000", 2023-11-01T04:16:54.447 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 31.368302077999999, 2023-11-01T04:16:54.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102610987, 2023-11-01T04:16:54.447 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.447 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.447 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.017682+0000", 2023-11-01T04:16:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.017682+0000", 2023-11-01T04:16:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.017683+0000", 2023-11-01T04:16:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6700000000000002e-07 2023-11-01T04:16:54.450 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.017731+0000", 2023-11-01T04:16:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7623999999999997e-05 2023-11-01T04:16:54.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.017765+0000", 2023-11-01T04:16:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4212999999999999e-05 2023-11-01T04:16:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.017790+0000", 2023-11-01T04:16:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4332e-05 2023-11-01T04:16:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.118102+0000", 2023-11-01T04:16:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100312568 2023-11-01T04:16:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.118128+0000", 2023-11-01T04:16:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6256000000000001e-05 2023-11-01T04:16:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.120214+0000", 2023-11-01T04:16:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002085642 2023-11-01T04:16:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:23.120293+0000", 2023-11-01T04:16:54.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9485000000000002e-05 2023-11-01T04:16:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 47/31 ECSubWrite(tid=52, reqid=client.4559.0:350, at_version=47'43, trim_to=0'0, roll_forward_to=45'41))", 2023-11-01T04:16:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.037415+0000", 2023-11-01T04:16:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.348569310999999, 2023-11-01T04:16:54.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10750568200000001, 2023-11-01T04:16:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.037415+0000", 2023-11-01T04:16:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.037415+0000", 2023-11-01T04:16:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.037416+0000", 2023-11-01T04:16:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.17e-06 2023-11-01T04:16:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.037474+0000", 2023-11-01T04:16:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7584999999999998e-05 2023-11-01T04:16:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.463 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.037507+0000", 2023-11-01T04:16:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3358999999999997e-05 2023-11-01T04:16:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.137630+0000", 2023-11-01T04:16:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100122251 2023-11-01T04:16:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.142712+0000", 2023-11-01T04:16:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0050826220000000002 2023-11-01T04:16:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.142732+0000", 2023-11-01T04:16:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9619999999999998e-05 2023-11-01T04:16:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.144845+0000", 2023-11-01T04:16:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021129040000000001 2023-11-01T04:16:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.144921+0000", 2023-11-01T04:16:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6171e-05 2023-11-01T04:16:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.469 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.6s6 47/31 ECSubWrite(tid=43, reqid=client.4559.0:351, at_version=47'6, trim_to=0'0, roll_forward_to=36'4))", 2023-11-01T04:16:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.248407+0000", 2023-11-01T04:16:54.470 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 27.137577630999999, 2023-11-01T04:16:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202387496, 2023-11-01T04:16:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248407+0000", 2023-11-01T04:16:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248407+0000", 2023-11-01T04:16:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248408+0000", 2023-11-01T04:16:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.037e-06 2023-11-01T04:16:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248462+0000", 2023-11-01T04:16:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4497999999999999e-05 2023-11-01T04:16:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248501+0000", 2023-11-01T04:16:54.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8423000000000001e-05 2023-11-01T04:16:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.348639+0000", 2023-11-01T04:16:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100138353 2023-11-01T04:16:54.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.448915+0000", 2023-11-01T04:16:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100276063 2023-11-01T04:16:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.448949+0000", 2023-11-01T04:16:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3482000000000002e-05 2023-11-01T04:16:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.450761+0000", 2023-11-01T04:16:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001811785 2023-11-01T04:16:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.450794+0000", 2023-11-01T04:16:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3855e-05 2023-11-01T04:16:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.480 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.1s2 47/31 ECSubWrite(tid=43, reqid=client.4559.0:355, at_version=47'23, trim_to=0'0, roll_forward_to=46'20))", 2023-11-01T04:16:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.484260+0000", 2023-11-01T04:16:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 26.901724140999999, 2023-11-01T04:16:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102572046, 2023-11-01T04:16:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.484260+0000", 2023-11-01T04:16:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.484260+0000", 2023-11-01T04:16:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.484262+0000", 2023-11-01T04:16:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1170000000000001e-06 2023-11-01T04:16:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.484450+0000", 2023-11-01T04:16:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018824 2023-11-01T04:16:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.484468+0000", 2023-11-01T04:16:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8084000000000002e-05 2023-11-01T04:16:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.484476+0000", 2023-11-01T04:16:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2309999999999996e-06 2023-11-01T04:16:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.584699+0000", 2023-11-01T04:16:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10022244700000001 2023-11-01T04:16:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.584716+0000", 2023-11-01T04:16:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7042999999999999e-05 2023-11-01T04:16:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.586754+0000", 2023-11-01T04:16:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020380929999999999 2023-11-01T04:16:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.586832+0000", 2023-11-01T04:16:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8790999999999998e-05 2023-11-01T04:16:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.as3 47/31 ECSubWrite(tid=77, reqid=client.4559.0:362, at_version=47'17, trim_to=0'0, roll_forward_to=44'15))", 2023-11-01T04:16:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.918704+0000", 2023-11-01T04:16:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 26.467280467999998, 2023-11-01T04:16:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10345299199999999, 2023-11-01T04:16:54.492 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.918704+0000", 2023-11-01T04:16:54.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.918704+0000", 2023-11-01T04:16:54.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.918705+0000", 2023-11-01T04:16:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5999999999999999e-07 2023-11-01T04:16:54.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.918936+0000", 2023-11-01T04:16:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023130000000000001 2023-11-01T04:16:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.918965+0000", 2023-11-01T04:16:54.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8745e-05 2023-11-01T04:16:54.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.918981+0000", 2023-11-01T04:16:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6529999999999999e-05 2023-11-01T04:16:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.019204+0000", 2023-11-01T04:16:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100222775 2023-11-01T04:16:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.019223+0000", 2023-11-01T04:16:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8958999999999999e-05 2023-11-01T04:16:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.022057+0000", 2023-11-01T04:16:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028342419999999998 2023-11-01T04:16:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.022157+0000", 2023-11-01T04:16:54.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9981000000000004e-05 2023-11-01T04:16:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.7s2 48/31 ECSubWrite(tid=46, reqid=client.4559.0:373, at_version=48'23, trim_to=0'0, roll_forward_to=46'20))", 2023-11-01T04:16:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:28.188861+0000", 2023-11-01T04:16:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 26.197123835999999, 2023-11-01T04:16:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20350378199999999, 2023-11-01T04:16:54.503 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.188861+0000", 2023-11-01T04:16:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.188861+0000", 2023-11-01T04:16:54.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.188862+0000", 2023-11-01T04:16:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2049999999999999e-06 2023-11-01T04:16:54.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.189188+0000", 2023-11-01T04:16:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00032597099999999999 2023-11-01T04:16:54.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.189217+0000", 2023-11-01T04:16:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9204999999999999e-05 2023-11-01T04:16:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.189237+0000", 2023-11-01T04:16:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.95e-05 2023-11-01T04:16:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.289496+0000", 2023-11-01T04:16:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100259104 2023-11-01T04:16:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.289526+0000", 2023-11-01T04:16:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9845e-05 2023-11-01T04:16:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.392321+0000", 2023-11-01T04:16:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10279562 2023-11-01T04:16:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.392364+0000", 2023-11-01T04:16:54.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3331999999999998e-05 2023-11-01T04:16:54.513 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.513 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.513 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.514 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.7s2 48/31 ECSubWrite(tid=47, reqid=client.4559.0:374, at_version=48'24, trim_to=0'0, roll_forward_to=46'20))", 2023-11-01T04:16:54.514 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:28.191530+0000", 2023-11-01T04:16:54.514 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 26.194454637, 2023-11-01T04:16:54.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.200865144, 2023-11-01T04:16:54.514 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.514 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.515 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.191530+0000", 2023-11-01T04:16:54.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.191530+0000", 2023-11-01T04:16:54.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.191531+0000", 2023-11-01T04:16:54.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9300000000000006e-07 2023-11-01T04:16:54.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.191574+0000", 2023-11-01T04:16:54.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3132e-05 2023-11-01T04:16:54.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.191588+0000", 2023-11-01T04:16:54.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4399000000000001e-05 2023-11-01T04:16:54.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.191596+0000", 2023-11-01T04:16:54.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3919999999999997e-06 2023-11-01T04:16:54.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.291060+0000", 2023-11-01T04:16:54.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.099463833000000001 2023-11-01T04:16:54.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.291088+0000", 2023-11-01T04:16:54.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8172999999999999e-05 2023-11-01T04:16:54.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.392372+0000", 2023-11-01T04:16:54.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101284208 2023-11-01T04:16:54.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.392395+0000", 2023-11-01T04:16:54.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3014e-05 2023-11-01T04:16:54.523 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.524 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.524 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.524 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.7s2 48/31 ECSubWrite(tid=48, reqid=client.4559.0:375, at_version=48'25, trim_to=0'0, roll_forward_to=46'20))", 2023-11-01T04:16:54.525 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:28.193820+0000", 2023-11-01T04:16:54.525 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 26.192164623, 2023-11-01T04:16:54.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19924911000000001, 2023-11-01T04:16:54.525 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.525 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.525 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.193820+0000", 2023-11-01T04:16:54.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.193820+0000", 2023-11-01T04:16:54.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.193821+0000", 2023-11-01T04:16:54.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0470000000000001e-06 2023-11-01T04:16:54.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.193842+0000", 2023-11-01T04:16:54.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0678000000000001e-05 2023-11-01T04:16:54.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.193858+0000", 2023-11-01T04:16:54.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6388e-05 2023-11-01T04:16:54.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.193867+0000", 2023-11-01T04:16:54.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.054e-06 2023-11-01T04:16:54.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.392049+0000", 2023-11-01T04:16:54.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.198182049 2023-11-01T04:16:54.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.392061+0000", 2023-11-01T04:16:54.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1687e-05 2023-11-01T04:16:54.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.392977+0000", 2023-11-01T04:16:54.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000916204 2023-11-01T04:16:54.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.393069+0000", 2023-11-01T04:16:54.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2003000000000007e-05 2023-11-01T04:16:54.534 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.534 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.534 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.535 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.6s6 54/31 ECSubWrite(tid=51, reqid=osd.14.0:51, at_version=54'8, trim_to=0'0, roll_forward_to=47'6))", 2023-11-01T04:16:54.535 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:34.105900+0000", 2023-11-01T04:16:54.535 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 20.280084814999999, 2023-11-01T04:16:54.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105133106, 2023-11-01T04:16:54.536 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.536 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.536 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.105900+0000", 2023-11-01T04:16:54.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.105900+0000", 2023-11-01T04:16:54.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.105900+0000", 2023-11-01T04:16:54.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4000000000000001e-07 2023-11-01T04:16:54.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.106017+0000", 2023-11-01T04:16:54.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117123 2023-11-01T04:16:54.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.106062+0000", 2023-11-01T04:16:54.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4536000000000002e-05 2023-11-01T04:16:54.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.106092+0000", 2023-11-01T04:16:54.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0375999999999999e-05 2023-11-01T04:16:54.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.206398+0000", 2023-11-01T04:16:54.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10030521100000001 2023-11-01T04:16:54.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.206425+0000", 2023-11-01T04:16:54.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7645999999999999e-05 2023-11-01T04:16:54.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.210955+0000", 2023-11-01T04:16:54.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045296499999999996 2023-11-01T04:16:54.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.211033+0000", 2023-11-01T04:16:54.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7923999999999995e-05 2023-11-01T04:16:54.545 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.545 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.545 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.546 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.1s2 57/31 ECSubWrite(tid=58, reqid=osd.6.0:58, at_version=57'34, trim_to=0'0, roll_forward_to=56'32))", 2023-11-01T04:16:54.546 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:37.442272+0000", 2023-11-01T04:16:54.546 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 16.943712986000001, 2023-11-01T04:16:54.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102805244, 2023-11-01T04:16:54.546 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.547 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.547 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.442272+0000", 2023-11-01T04:16:54.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.442272+0000", 2023-11-01T04:16:54.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.442272+0000", 2023-11-01T04:16:54.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4000000000000001e-07 2023-11-01T04:16:54.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.442326+0000", 2023-11-01T04:16:54.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3461999999999997e-05 2023-11-01T04:16:54.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.442363+0000", 2023-11-01T04:16:54.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7598999999999997e-05 2023-11-01T04:16:54.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.542467+0000", 2023-11-01T04:16:54.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100103435 2023-11-01T04:16:54.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.542650+0000", 2023-11-01T04:16:54.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000183499 2023-11-01T04:16:54.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.542677+0000", 2023-11-01T04:16:54.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6556000000000001e-05 2023-11-01T04:16:54.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.544983+0000", 2023-11-01T04:16:54.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023058789999999998 2023-11-01T04:16:54.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.545077+0000", 2023-11-01T04:16:54.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4173999999999997e-05 2023-11-01T04:16:54.555 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.556 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.556 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.556 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.7s2 58/31 ECSubWrite(tid=65, reqid=osd.14.0:65, at_version=58'38, trim_to=0'0, roll_forward_to=57'36))", 2023-11-01T04:16:54.557 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:38.452520+0000", 2023-11-01T04:16:54.557 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 15.933464807, 2023-11-01T04:16:54.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202505307, 2023-11-01T04:16:54.557 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.557 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.558 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.452520+0000", 2023-11-01T04:16:54.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.452520+0000", 2023-11-01T04:16:54.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.452520+0000", 2023-11-01T04:16:54.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1600000000000002e-07 2023-11-01T04:16:54.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.452553+0000", 2023-11-01T04:16:54.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3322999999999999e-05 2023-11-01T04:16:54.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.452581+0000", 2023-11-01T04:16:54.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7665999999999999e-05 2023-11-01T04:16:54.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.452597+0000", 2023-11-01T04:16:54.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6413e-05 2023-11-01T04:16:54.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.652927+0000", 2023-11-01T04:16:54.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20032907899999999 2023-11-01T04:16:54.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.652945+0000", 2023-11-01T04:16:54.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8896e-05 2023-11-01T04:16:54.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.654953+0000", 2023-11-01T04:16:54.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020074759999999998 2023-11-01T04:16:54.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.655025+0000", 2023-11-01T04:16:54.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2138000000000006e-05 2023-11-01T04:16:54.567 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.567 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.567 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.568 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.bs3 62/31 ECSubWrite(tid=70, reqid=client.4559.0:479, at_version=62'17, trim_to=0'0, roll_forward_to=45'15))", 2023-11-01T04:16:54.568 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:42.297728+0000", 2023-11-01T04:16:54.568 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 12.088256688, 2023-11-01T04:16:54.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10443308799999999, 2023-11-01T04:16:54.568 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.569 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.569 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.297728+0000", 2023-11-01T04:16:54.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.297728+0000", 2023-11-01T04:16:54.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.297729+0000", 2023-11-01T04:16:54.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1699999999999997e-07 2023-11-01T04:16:54.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.298270+0000", 2023-11-01T04:16:54.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00054102399999999998 2023-11-01T04:16:54.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.298303+0000", 2023-11-01T04:16:54.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3785999999999998e-05 2023-11-01T04:16:54.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.298331+0000", 2023-11-01T04:16:54.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7330000000000001e-05 2023-11-01T04:16:54.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.398610+0000", 2023-11-01T04:16:54.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100279405 2023-11-01T04:16:54.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.398627+0000", 2023-11-01T04:16:54.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6685999999999999e-05 2023-11-01T04:16:54.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.402088+0000", 2023-11-01T04:16:54.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034616550000000001 2023-11-01T04:16:54.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.402161+0000", 2023-11-01T04:16:54.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2484999999999995e-05 2023-11-01T04:16:54.578 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.578 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.578 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.579 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.as3 67/31 ECSubWrite(tid=131, reqid=client.4559.0:535, at_version=0'0, trim_to=0'0, roll_forward_to=67'30))", 2023-11-01T04:16:54.579 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.392875+0000", 2023-11-01T04:16:54.579 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 6.9931095179999998, 2023-11-01T04:16:54.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20174935799999999, 2023-11-01T04:16:54.580 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.580 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.580 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.392875+0000", 2023-11-01T04:16:54.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.392875+0000", 2023-11-01T04:16:54.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.392878+0000", 2023-11-01T04:16:54.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.88e-06 2023-11-01T04:16:54.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.392907+0000", 2023-11-01T04:16:54.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8693000000000001e-05 2023-11-01T04:16:54.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.392954+0000", 2023-11-01T04:16:54.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7688999999999997e-05 2023-11-01T04:16:54.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.493051+0000", 2023-11-01T04:16:54.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10009689200000001 2023-11-01T04:16:54.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.593262+0000", 2023-11-01T04:16:54.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10021129400000001 2023-11-01T04:16:54.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.593281+0000", 2023-11-01T04:16:54.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8695999999999999e-05 2023-11-01T04:16:54.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.594577+0000", 2023-11-01T04:16:54.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012961680000000001 2023-11-01T04:16:54.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.594624+0000", 2023-11-01T04:16:54.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7046000000000003e-05 2023-11-01T04:16:54.589 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.589 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.589 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.590 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 67/31 ECSubWrite(tid=77, reqid=client.4559.0:537, at_version=67'58, trim_to=0'0, roll_forward_to=67'56))", 2023-11-01T04:16:54.590 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.705899+0000", 2023-11-01T04:16:54.590 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 6.6800855779999999, 2023-11-01T04:16:54.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20538421000000001, 2023-11-01T04:16:54.590 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.590 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.591 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.705899+0000", 2023-11-01T04:16:54.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.705899+0000", 2023-11-01T04:16:54.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.705900+0000", 2023-11-01T04:16:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5200000000000001e-06 2023-11-01T04:16:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.706759+0000", 2023-11-01T04:16:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00085813999999999997 2023-11-01T04:16:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.706787+0000", 2023-11-01T04:16:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8385999999999999e-05 2023-11-01T04:16:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.806881+0000", 2023-11-01T04:16:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10009428200000001 2023-11-01T04:16:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.907197+0000", 2023-11-01T04:16:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100315683 2023-11-01T04:16:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.907230+0000", 2023-11-01T04:16:54.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3464999999999999e-05 2023-11-01T04:16:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.911219+0000", 2023-11-01T04:16:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039884150000000004 2023-11-01T04:16:54.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.911283+0000", 2023-11-01T04:16:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4319000000000005e-05 2023-11-01T04:16:54.599 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.600 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.7s2 68/31 ECSubWrite(tid=83, reqid=client.4559.0:546, at_version=68'44, trim_to=0'0, roll_forward_to=67'42))", 2023-11-01T04:16:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.893994+0000", 2023-11-01T04:16:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 6.4919906709999999, 2023-11-01T04:16:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10444176500000001, 2023-11-01T04:16:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:54.601 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.893994+0000", 2023-11-01T04:16:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.893994+0000", 2023-11-01T04:16:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.893996+0000", 2023-11-01T04:16:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9769999999999999e-06 2023-11-01T04:16:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894238+0000", 2023-11-01T04:16:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024247900000000001 2023-11-01T04:16:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:54.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894281+0000", 2023-11-01T04:16:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2392999999999999e-05 2023-11-01T04:16:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:54.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894303+0000", 2023-11-01T04:16:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2612e-05 2023-11-01T04:16:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.994602+0000", 2023-11-01T04:16:54.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100298312 2023-11-01T04:16:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.994621+0000", 2023-11-01T04:16:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.893e-05 2023-11-01T04:16:54.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.998363+0000", 2023-11-01T04:16:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037421260000000001 2023-11-01T04:16:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:54.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.998436+0000", 2023-11-01T04:16:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2935999999999994e-05 2023-11-01T04:16:54.610 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:54.611 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:54.611 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:54.612 INFO:tasks.rados.rados.0.smithi190.stdout:249: finishing write tid 4 to smithi19039307-5 2023-11-01T04:16:54.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-01T04:16:54.614 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:54.611+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:54.714 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:54.711+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:54.743 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:54.743 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:54.743 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2023-11-01T04:16:54.744 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:54.757 INFO:teuthology.orchestra.run.smithi153.stderr:noscrub is set 2023-11-01T04:16:54.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-01T04:16:54.764 INFO:tasks.rados.rados.0.smithi190.stdout:252: done (2 left) 2023-11-01T04:16:54.764 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 23 current snap is 23 2023-11-01T04:16:54.764 INFO:tasks.rados.rados.0.smithi190.stdout:253: seq_num 104 ranges {0=589824,589824=540672,1130496=622592} 2023-11-01T04:16:54.772 INFO:tasks.rados.rados.0.smithi190.stdout:253: writing smithi19039307-23 from 0 to 589824 tid 1 2023-11-01T04:16:54.780 INFO:tasks.rados.rados.0.smithi190.stdout:253: writing smithi19039307-23 from 589824 to 1130496 tid 2 2023-11-01T04:16:54.788 INFO:tasks.rados.rados.0.smithi190.stdout:253: writing smithi19039307-23 from 1130496 to 1753088 tid 3 2023-11-01T04:16:54.788 INFO:tasks.rados.rados.0.smithi190.stdout:254: rollback oid 26 current snap is 23 2023-11-01T04:16:54.789 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 26 to 12 2023-11-01T04:16:54.789 INFO:tasks.rados.rados.0.smithi190.stdout:255: rmattr oid 22 current snap is 23 2023-11-01T04:16:54.789 INFO:tasks.rados.rados.0.smithi190.stdout:255: done (4 left) 2023-11-01T04:16:54.789 INFO:tasks.rados.rados.0.smithi190.stdout:256: delete oid 29 current snap is 23 2023-11-01T04:16:54.814 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:16:54.876 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:54.876 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:16:54.876 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:16:54.876 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:16:54.877 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:54.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-01T04:16:54.894 INFO:tasks.rados.rados.0.smithi190.stdout:250: finishing copy_from to smithi19039307-48 2023-11-01T04:16:54.894 INFO:tasks.rados.rados.0.smithi190.stdout:250: got expected ENOENT (src dne) 2023-11-01T04:16:54.895 INFO:tasks.rados.rados.0.smithi190.stdout:250: finishing copy_from racing read to smithi19039307-48 2023-11-01T04:16:54.916 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:16:54.914+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:16:54.938 INFO:tasks.rados.rados.0.smithi190.stdout:249: finishing write tid 5 to smithi19039307-5 2023-11-01T04:16:54.938 INFO:tasks.rados.rados.0.smithi190.stdout:249: oid 5 updating version 0 to 64 2023-11-01T04:16:54.938 INFO:tasks.rados.rados.0.smithi190.stdout:249: oid 5 version 64 is already newer than 62 2023-11-01T04:16:54.939 INFO:tasks.rados.rados.0.smithi190.stdout:249: oid 5 version 64 is already newer than 63 2023-11-01T04:16:54.939 INFO:tasks.rados.rados.0.smithi190.stdout:249: oid 5 version 64 is already newer than 61 2023-11-01T04:16:54.939 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 5 v 64 (ObjNum 103 snap 21 seq_num 103) dirty exists 2023-11-01T04:16:54.939 INFO:tasks.rados.rados.0.smithi190.stdout:249: left oid 5 (ObjNum 103 snap 21 seq_num 103) 2023-11-01T04:16:55.011 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:16:55.011 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:16:55.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:16:55.011 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:16:55.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.012 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:55.012 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138761+0000", 2023-11-01T04:16:55.012 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 49.871243307, 2023-11-01T04:16:55.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61681865999999996, 2023-11-01T04:16:55.013 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.013 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:55.013 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.013 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:55.013 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:55.014 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 114 2023-11-01T04:16:55.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.014 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138761+0000", 2023-11-01T04:16:55.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138761+0000", 2023-11-01T04:16:55.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138762+0000", 2023-11-01T04:16:55.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4300000000000003e-07 2023-11-01T04:16:55.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138768+0000", 2023-11-01T04:16:55.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7729999999999996e-06 2023-11-01T04:16:55.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138777+0000", 2023-11-01T04:16:55.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1929999999999999e-06 2023-11-01T04:16:55.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138784+0000", 2023-11-01T04:16:55.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3830000000000003e-06 2023-11-01T04:16:55.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.248094+0000", 2023-11-01T04:16:55.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10930951699999999 2023-11-01T04:16:55.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.248210+0000", 2023-11-01T04:16:55.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011579299999999999 2023-11-01T04:16:55.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:55.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.248407+0000", 2023-11-01T04:16:55.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019723000000000001 2023-11-01T04:16:55.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:55.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352490+0000", 2023-11-01T04:16:55.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104083393 2023-11-01T04:16:55.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:55.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755390+0000", 2023-11-01T04:16:55.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40290009500000001 2023-11-01T04:16:55.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:55.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755474+0000", 2023-11-01T04:16:55.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3323000000000002e-05 2023-11-01T04:16:55.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755580+0000", 2023-11-01T04:16:55.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010631700000000001 2023-11-01T04:16:55.029 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.029 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.029 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.030 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:55.030 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138795+0000", 2023-11-01T04:16:55.031 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 49.871209149000002, 2023-11-01T04:16:55.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61707305000000001, 2023-11-01T04:16:55.031 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.031 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:55.031 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.032 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:55.032 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:55.032 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 115 2023-11-01T04:16:55.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.033 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138795+0000", 2023-11-01T04:16:55.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138795+0000", 2023-11-01T04:16:55.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138796+0000", 2023-11-01T04:16:55.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3300000000000002e-07 2023-11-01T04:16:55.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138931+0000", 2023-11-01T04:16:55.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013525000000000001 2023-11-01T04:16:55.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138947+0000", 2023-11-01T04:16:55.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5478999999999999e-05 2023-11-01T04:16:55.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138956+0000", 2023-11-01T04:16:55.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1530000000000003e-06 2023-11-01T04:16:55.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.249005+0000", 2023-11-01T04:16:55.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.110048914 2023-11-01T04:16:55.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.249150+0000", 2023-11-01T04:16:55.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014491600000000001 2023-11-01T04:16:55.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:55.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.249351+0000", 2023-11-01T04:16:55.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000201516 2023-11-01T04:16:55.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:55.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352738+0000", 2023-11-01T04:16:55.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10338632 2023-11-01T04:16:55.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:55.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755668+0000", 2023-11-01T04:16:55.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40293035700000002 2023-11-01T04:16:55.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:55.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755765+0000", 2023-11-01T04:16:55.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7081000000000001e-05 2023-11-01T04:16:55.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755868+0000", 2023-11-01T04:16:55.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010343100000000001 2023-11-01T04:16:55.046 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.046 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.046 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.047 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:55.047 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138971+0000", 2023-11-01T04:16:55.047 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 49.871033611000001, 2023-11-01T04:16:55.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71733164100000002, 2023-11-01T04:16:55.048 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.048 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:55.048 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.048 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:55.049 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:55.049 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 116 2023-11-01T04:16:55.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.049 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138971+0000", 2023-11-01T04:16:55.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138971+0000", 2023-11-01T04:16:55.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138972+0000", 2023-11-01T04:16:55.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4000000000000001e-07 2023-11-01T04:16:55.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138988+0000", 2023-11-01T04:16:55.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6354999999999998e-05 2023-11-01T04:16:55.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138998+0000", 2023-11-01T04:16:55.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0285999999999999e-05 2023-11-01T04:16:55.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139010+0000", 2023-11-01T04:16:55.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1807000000000001e-05 2023-11-01T04:16:55.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.350520+0000", 2023-11-01T04:16:55.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21151004200000001 2023-11-01T04:16:55.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.350668+0000", 2023-11-01T04:16:55.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014824900000000001 2023-11-01T04:16:55.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:55.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.350873+0000", 2023-11-01T04:16:55.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020507899999999999 2023-11-01T04:16:55.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:55.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352744+0000", 2023-11-01T04:16:55.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001870741 2023-11-01T04:16:55.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:55.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856134+0000", 2023-11-01T04:16:55.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50339015399999998 2023-11-01T04:16:55.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:55.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856235+0000", 2023-11-01T04:16:55.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010087200000000001 2023-11-01T04:16:55.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856303+0000", 2023-11-01T04:16:55.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7316000000000004e-05 2023-11-01T04:16:55.064 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.064 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.064 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.065 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:55.065 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.139035+0000", 2023-11-01T04:16:55.065 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 49.870969625000001, 2023-11-01T04:16:55.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71770837499999995, 2023-11-01T04:16:55.066 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.066 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:55.066 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.066 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:55.066 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:55.067 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 117 2023-11-01T04:16:55.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.067 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139035+0000", 2023-11-01T04:16:55.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139035+0000", 2023-11-01T04:16:55.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139036+0000", 2023-11-01T04:16:55.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7999999999999995e-07 2023-11-01T04:16:55.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139042+0000", 2023-11-01T04:16:55.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9499999999999998e-06 2023-11-01T04:16:55.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139051+0000", 2023-11-01T04:16:55.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7489999999999993e-06 2023-11-01T04:16:55.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139059+0000", 2023-11-01T04:16:55.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8159999999999994e-06 2023-11-01T04:16:55.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.351436+0000", 2023-11-01T04:16:55.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21237641400000001 2023-11-01T04:16:55.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.351583+0000", 2023-11-01T04:16:55.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000147186 2023-11-01T04:16:55.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:55.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.351772+0000", 2023-11-01T04:16:55.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018879300000000001 2023-11-01T04:16:55.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:55.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.553963+0000", 2023-11-01T04:16:55.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20219130599999999 2023-11-01T04:16:55.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:55.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856458+0000", 2023-11-01T04:16:55.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30249546300000002 2023-11-01T04:16:55.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:55.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856632+0000", 2023-11-01T04:16:55.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000173611 2023-11-01T04:16:55.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856743+0000", 2023-11-01T04:16:55.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111407 2023-11-01T04:16:55.080 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.081 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.081 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.081 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:55.082 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622627+0000", 2023-11-01T04:16:55.082 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 48.387377925999999, 2023-11-01T04:16:55.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60704219400000003, 2023-11-01T04:16:55.082 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.082 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:55.083 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.083 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:55.083 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:55.083 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 139 2023-11-01T04:16:55.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.084 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622627+0000", 2023-11-01T04:16:55.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622627+0000", 2023-11-01T04:16:55.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622627+0000", 2023-11-01T04:16:55.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5400000000000001e-07 2023-11-01T04:16:55.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622632+0000", 2023-11-01T04:16:55.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0420000000000002e-06 2023-11-01T04:16:55.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622642+0000", 2023-11-01T04:16:55.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.713e-06 2023-11-01T04:16:55.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622649+0000", 2023-11-01T04:16:55.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4460000000000004e-06 2023-11-01T04:16:55.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.831442+0000", 2023-11-01T04:16:55.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20879259 2023-11-01T04:16:55.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.831689+0000", 2023-11-01T04:16:55.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000247087 2023-11-01T04:16:55.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:55.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.832051+0000", 2023-11-01T04:16:55.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036175699999999999 2023-11-01T04:16:55.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:55.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.837325+0000", 2023-11-01T04:16:55.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0052741289999999998 2023-11-01T04:16:55.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:55.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229409+0000", 2023-11-01T04:16:55.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39208416000000001 2023-11-01T04:16:55.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:55.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229566+0000", 2023-11-01T04:16:55.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015727499999999999 2023-11-01T04:16:55.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229669+0000", 2023-11-01T04:16:55.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000102341 2023-11-01T04:16:55.097 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.098 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.098 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.099 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:55.099 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622661+0000", 2023-11-01T04:16:55.099 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 48.387343534000003, 2023-11-01T04:16:55.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60735416200000003, 2023-11-01T04:16:55.099 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.099 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:55.100 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.100 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:55.100 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:55.100 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 140 2023-11-01T04:16:55.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.101 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622661+0000", 2023-11-01T04:16:55.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622661+0000", 2023-11-01T04:16:55.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622662+0000", 2023-11-01T04:16:55.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3e-07 2023-11-01T04:16:55.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622803+0000", 2023-11-01T04:16:55.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014142199999999999 2023-11-01T04:16:55.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622816+0000", 2023-11-01T04:16:55.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2683e-05 2023-11-01T04:16:55.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622825+0000", 2023-11-01T04:16:55.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9959999999999999e-06 2023-11-01T04:16:55.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.833052+0000", 2023-11-01T04:16:55.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21022711699999999 2023-11-01T04:16:55.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.833201+0000", 2023-11-01T04:16:55.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000149059 2023-11-01T04:16:55.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:55.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.833405+0000", 2023-11-01T04:16:55.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020363600000000001 2023-11-01T04:16:55.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:55.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.837343+0000", 2023-11-01T04:16:55.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039379590000000004 2023-11-01T04:16:55.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:55.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229871+0000", 2023-11-01T04:16:55.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39252865399999998 2023-11-01T04:16:55.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:55.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229957+0000", 2023-11-01T04:16:55.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5962000000000001e-05 2023-11-01T04:16:55.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.230015+0000", 2023-11-01T04:16:55.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8044000000000002e-05 2023-11-01T04:16:55.114 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.115 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.115 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.115 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:16:55.116 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622843+0000", 2023-11-01T04:16:55.116 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 48.387161208000002, 2023-11-01T04:16:55.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70733220600000002, 2023-11-01T04:16:55.116 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.116 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:55.117 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.117 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:16:55.117 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:16:55.117 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 141 2023-11-01T04:16:55.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.118 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622843+0000", 2023-11-01T04:16:55.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622843+0000", 2023-11-01T04:16:55.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622844+0000", 2023-11-01T04:16:55.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7400000000000002e-07 2023-11-01T04:16:55.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622850+0000", 2023-11-01T04:16:55.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4330000000000003e-06 2023-11-01T04:16:55.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622859+0000", 2023-11-01T04:16:55.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5829999999999996e-06 2023-11-01T04:16:55.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.722958+0000", 2023-11-01T04:16:55.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100098824 2023-11-01T04:16:55.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835872+0000", 2023-11-01T04:16:55.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.112913586 2023-11-01T04:16:55.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.836107+0000", 2023-11-01T04:16:55.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023507399999999999 2023-11-01T04:16:55.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:16:55.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.836474+0000", 2023-11-01T04:16:55.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036769800000000002 2023-11-01T04:16:55.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:16:55.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.837993+0000", 2023-11-01T04:16:55.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001518192 2023-11-01T04:16:55.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:16:55.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.329913+0000", 2023-11-01T04:16:55.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.49192055499999998 2023-11-01T04:16:55.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:55.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.330067+0000", 2023-11-01T04:16:55.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000154012 2023-11-01T04:16:55.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.330176+0000", 2023-11-01T04:16:55.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010847499999999999 2023-11-01T04:16:55.132 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.132 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.132 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.133 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:207 3.2s0 3:42ba522a:::smithi19039307-20:head [copy-from ver 4] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:55.133 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.965579+0000", 2023-11-01T04:16:55.133 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.044425212999997, 2023-11-01T04:16:55.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62034388299999998, 2023-11-01T04:16:55.133 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.134 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:55.134 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.134 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:55.134 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:55.135 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 207 2023-11-01T04:16:55.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.135 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965579+0000", 2023-11-01T04:16:55.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965579+0000", 2023-11-01T04:16:55.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965581+0000", 2023-11-01T04:16:55.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1969999999999999e-06 2023-11-01T04:16:55.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965617+0000", 2023-11-01T04:16:55.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6100999999999999e-05 2023-11-01T04:16:55.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965640+0000", 2023-11-01T04:16:55.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3566000000000001e-05 2023-11-01T04:16:55.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965657+0000", 2023-11-01T04:16:55.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6413e-05 2023-11-01T04:16:55.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.066185+0000", 2023-11-01T04:16:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100528453 2023-11-01T04:16:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.066331+0000", 2023-11-01T04:16:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014583099999999999 2023-11-01T04:16:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.381227+0000", 2023-11-01T04:16:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31489625199999999 2023-11-01T04:16:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.384145+0000", 2023-11-01T04:16:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029175310000000001 2023-11-01T04:16:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:55.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.585865+0000", 2023-11-01T04:16:55.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.201720436 2023-11-01T04:16:55.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.585923+0000", 2023-11-01T04:16:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8103000000000003e-05 2023-11-01T04:16:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.147 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:209 3.3s0 3:ca2966a4:::smithi19039307-37:head [read 0~606208,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.966335+0000", 2023-11-01T04:16:55.148 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.043669352999999, 2023-11-01T04:16:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61218852199999996, 2023-11-01T04:16:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.149 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 209 2023-11-01T04:16:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966335+0000", 2023-11-01T04:16:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966335+0000", 2023-11-01T04:16:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966337+0000", 2023-11-01T04:16:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1430000000000001e-06 2023-11-01T04:16:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966349+0000", 2023-11-01T04:16:55.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2053e-05 2023-11-01T04:16:55.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966385+0000", 2023-11-01T04:16:55.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5438999999999997e-05 2023-11-01T04:16:55.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966408+0000", 2023-11-01T04:16:55.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3349e-05 2023-11-01T04:16:55.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970845+0000", 2023-11-01T04:16:55.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044366639999999999 2023-11-01T04:16:55.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970977+0000", 2023-11-01T04:16:55.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013261100000000001 2023-11-01T04:16:55.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.578524+0000", 2023-11-01T04:16:55.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60754626300000003 2023-11-01T04:16:55.159 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.159 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.159 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.160 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:210 3.3s0 3:ca2966a4:::smithi19039307-37:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:55.160 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.966473+0000", 2023-11-01T04:16:55.161 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.043531666, 2023-11-01T04:16:55.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61269195099999996, 2023-11-01T04:16:55.161 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.161 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:55.161 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.162 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:55.162 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:55.162 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 210 2023-11-01T04:16:55.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.162 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966473+0000", 2023-11-01T04:16:55.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966473+0000", 2023-11-01T04:16:55.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966474+0000", 2023-11-01T04:16:55.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4500000000000001e-06 2023-11-01T04:16:55.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966484+0000", 2023-11-01T04:16:55.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2269999999999999e-06 2023-11-01T04:16:55.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966508+0000", 2023-11-01T04:16:55.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4596e-05 2023-11-01T04:16:55.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966526+0000", 2023-11-01T04:16:55.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7938999999999999e-05 2023-11-01T04:16:55.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.071473+0000", 2023-11-01T04:16:55.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10494663799999999 2023-11-01T04:16:55.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.071640+0000", 2023-11-01T04:16:55.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016699 2023-11-01T04:16:55.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.579165+0000", 2023-11-01T04:16:55.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50752511099999997 2023-11-01T04:16:55.172 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.172 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.172 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.173 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:211 3.3s0 3:ca2966a4:::smithi19039307-37:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:55.173 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.966567+0000", 2023-11-01T04:16:55.173 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.043438029000001, 2023-11-01T04:16:55.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61304655900000005, 2023-11-01T04:16:55.174 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.174 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:55.174 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.174 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:55.175 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:55.175 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 211 2023-11-01T04:16:55.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.175 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966567+0000", 2023-11-01T04:16:55.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966567+0000", 2023-11-01T04:16:55.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966568+0000", 2023-11-01T04:16:55.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2529999999999999e-06 2023-11-01T04:16:55.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966575+0000", 2023-11-01T04:16:55.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4030000000000001e-06 2023-11-01T04:16:55.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966595+0000", 2023-11-01T04:16:55.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0208999999999999e-05 2023-11-01T04:16:55.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966611+0000", 2023-11-01T04:16:55.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6082999999999998e-05 2023-11-01T04:16:55.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.072213+0000", 2023-11-01T04:16:55.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105601509 2023-11-01T04:16:55.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.072432+0000", 2023-11-01T04:16:55.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000218724 2023-11-01T04:16:55.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.579613+0000", 2023-11-01T04:16:55.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50718137799999996 2023-11-01T04:16:55.184 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.184 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.184 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.185 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:212 3.3s0 3:c87735c0:::smithi19039307-50:head [create,setxattr XNeZS[lOPtVca\\Pa[\\rdfbinb[bcjhtYSpmcafZm (1928) in=40b,setxattr TfflYYqPnps]lo^qSlp___TitNdd`]kaPTVOTcsh (1648) in=40b,setxattr TNb_onpY[j_dkhfmnWtm]Yktg]]RNtbQe[OUTNlX (770) in=40b,setxattr ZcgnkhY\\orTYaZh]PPldj`SNq`p\\cX`k]h\\bZY_Y (700) in=40b,setxattr gPQVnqhpsVhW_kq`QnafXrlRZtZ\\YnRftgQTishR (595) in=40b,setxattr U`XqYi[WhoqnpSPRc\\RhnppZp^qY\\qUfjpqs[r^a (1892) in=40b,setxattr cTmhd`pWgrRf_VfVgOtgeOP^ShP]lsOtpoTj[\\gl (418) in=40b,setxattr aa_taYTfslg`lRRrOkkraN\\XiW^V\\ljl]NqXin`a (314) in=40b,setxattr ibl]`igtgVThY^Qse\\okYYUtTmolX\\W]p]Q[P\\il (1403) in=40b,setxattr \\ZabSO[UUqeZa`hddNh[ZjaRRgUZoQtkn\\QdsX]W (1541) in=40b,setxattr jUkQUPTNbo`rZNmYlUmhb]ojc]aeheoVRfp]bPWf (5052) in=40b,setxattr Ofj`oQ^cR]da^SPsTaXmmPse\\OZ`VSqQ]U`efsZl (419) in=40b,setxattr a[cdkWos\\OrUVPciUaa^_^`lWVTScffiWkd\\[kgf (1261) in=40b,setxattr jXb]\\WhbatdZ[qTTiZO]kPd^`l[kajOYiQ[iqSYc (244) in=40b,setxattr \\im]jQhbanUn\\_^h`jaU]W`ReaZN_je[]kgg (1578) in=36b,setxattr ZbosoRORS^_de]rN^_daOiha\\Q\\eRae^[sQUjXsV (9120) in=40b,setxattr gcn]XpZUkOZr\\pkrVVZpYiR\\alh_PaZa_S[PYcn] (384) in=40b,setxattr \\^qUOaNXSTjtmafhfbo\\gseTTkV`mgnacYq^bkcc (34) in=40b,setxattr iUa^mOQ]TqjVSUU_nXfY]f`NjljiiaYbrm[o]aes (1611) in=40b,setxattr tXaQPQbc^aiV[^reVOPpib^caoWf^XgqilO^_g[k (1016) in=40b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:55.185 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.970354+0000", 2023-11-01T04:16:55.185 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.039650420000001, 2023-11-01T04:16:55.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91502122100000005, 2023-11-01T04:16:55.186 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.186 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:55.186 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.186 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:55.186 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:55.187 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 212 2023-11-01T04:16:55.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.187 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970354+0000", 2023-11-01T04:16:55.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970354+0000", 2023-11-01T04:16:55.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970356+0000", 2023-11-01T04:16:55.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3070000000000001e-06 2023-11-01T04:16:55.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970468+0000", 2023-11-01T04:16:55.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111496 2023-11-01T04:16:55.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970492+0000", 2023-11-01T04:16:55.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4096000000000001e-05 2023-11-01T04:16:55.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970511+0000", 2023-11-01T04:16:55.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9069e-05 2023-11-01T04:16:55.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.579670+0000", 2023-11-01T04:16:55.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60915843300000005 2023-11-01T04:16:55.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.579779+0000", 2023-11-01T04:16:55.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000109417 2023-11-01T04:16:55.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.580741+0000", 2023-11-01T04:16:55.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00096157299999999996 2023-11-01T04:16:55.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.582542+0000", 2023-11-01T04:16:55.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018016130000000001 2023-11-01T04:16:55.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:55.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.885257+0000", 2023-11-01T04:16:55.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30271534300000003 2023-11-01T04:16:55.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.885375+0000", 2023-11-01T04:16:55.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117874 2023-11-01T04:16:55.200 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.200 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.200 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.201 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:496 3.2s0 3:4174a5c9:::smithi19039307-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2023-11-01T04:16:55.201 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:42.354747+0000", 2023-11-01T04:16:55.201 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 12.65525794, 2023-11-01T04:16:55.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61943324300000002, 2023-11-01T04:16:55.202 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.202 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:55.202 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.202 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:55.202 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:55.203 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 496 2023-11-01T04:16:55.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.203 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354747+0000", 2023-11-01T04:16:55.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354747+0000", 2023-11-01T04:16:55.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354747+0000", 2023-11-01T04:16:55.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0299999999999998e-07 2023-11-01T04:16:55.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354751+0000", 2023-11-01T04:16:55.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.326e-06 2023-11-01T04:16:55.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354763+0000", 2023-11-01T04:16:55.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2716e-05 2023-11-01T04:16:55.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354772+0000", 2023-11-01T04:16:55.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4570000000000005e-06 2023-11-01T04:16:55.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.357535+0000", 2023-11-01T04:16:55.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002763148 2023-11-01T04:16:55.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:55.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.357552+0000", 2023-11-01T04:16:55.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6849000000000001e-05 2023-11-01T04:16:55.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.568063+0000", 2023-11-01T04:16:55.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.210510788 2023-11-01T04:16:55.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2023-11-01T04:16:55.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.568200+0000", 2023-11-01T04:16:55.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013761000000000001 2023-11-01T04:16:55.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.670978+0000", 2023-11-01T04:16:55.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102777572 2023-11-01T04:16:55.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.671121+0000", 2023-11-01T04:16:55.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014276299999999999 2023-11-01T04:16:55.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.974180+0000", 2023-11-01T04:16:55.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.303059211 2023-11-01T04:16:55.216 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.216 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.217 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.217 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:552 3.2s0 3:42f00163:::smithi19039307-2:head [copy-from ver 12] snapc 1e=[1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:16:55.217 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.820369+0000", 2023-11-01T04:16:55.217 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 7.1896356959999999, 2023-11-01T04:16:55.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89219467399999997, 2023-11-01T04:16:55.218 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.218 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:55.218 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.219 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:55.219 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:55.219 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 552 2023-11-01T04:16:55.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.219 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820369+0000", 2023-11-01T04:16:55.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820369+0000", 2023-11-01T04:16:55.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820371+0000", 2023-11-01T04:16:55.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.266e-06 2023-11-01T04:16:55.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820383+0000", 2023-11-01T04:16:55.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2023e-05 2023-11-01T04:16:55.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820415+0000", 2023-11-01T04:16:55.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2255999999999997e-05 2023-11-01T04:16:55.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820431+0000", 2023-11-01T04:16:55.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5823000000000001e-05 2023-11-01T04:16:55.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.079379+0000", 2023-11-01T04:16:55.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25894801099999998 2023-11-01T04:16:55.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.079550+0000", 2023-11-01T04:16:55.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017098600000000001 2023-11-01T04:16:55.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.494335+0000", 2023-11-01T04:16:55.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41478489200000002 2023-11-01T04:16:55.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.498143+0000", 2023-11-01T04:16:55.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038076920000000001 2023-11-01T04:16:55.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:55.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.712428+0000", 2023-11-01T04:16:55.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21428475699999999 2023-11-01T04:16:55.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.712564+0000", 2023-11-01T04:16:55.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013596800000000001 2023-11-01T04:16:55.231 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.232 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.232 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.233 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:553 3.2s0 3:42f00163:::smithi19039307-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:16:55.233 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.820458+0000", 2023-11-01T04:16:55.233 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 7.1895463409999998, 2023-11-01T04:16:55.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.67673522200000003, 2023-11-01T04:16:55.233 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.234 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:55.234 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.234 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:55.234 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:55.235 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 553 2023-11-01T04:16:55.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.235 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820458+0000", 2023-11-01T04:16:55.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820458+0000", 2023-11-01T04:16:55.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820459+0000", 2023-11-01T04:16:55.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8299999999999995e-07 2023-11-01T04:16:55.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820463+0000", 2023-11-01T04:16:55.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9369999999999997e-06 2023-11-01T04:16:55.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820475+0000", 2023-11-01T04:16:55.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2026e-05 2023-11-01T04:16:55.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.920578+0000", 2023-11-01T04:16:55.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10010329 2023-11-01T04:16:55.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.283349+0000", 2023-11-01T04:16:55.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36277072599999999 2023-11-01T04:16:55.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for blocked object", 2023-11-01T04:16:55.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.283446+0000", 2023-11-01T04:16:55.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7083999999999994e-05 2023-11-01T04:16:55.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.496890+0000", 2023-11-01T04:16:55.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21344336899999999 2023-11-01T04:16:55.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.497036+0000", 2023-11-01T04:16:55.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000146324 2023-11-01T04:16:55.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.497193+0000", 2023-11-01T04:16:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015758300000000001 2023-11-01T04:16:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:565 3.2s0 3:42f00163:::smithi19039307-2:head [read 0~557056,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2023-11-01T04:16:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:49.511279+0000", 2023-11-01T04:16:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.4987250630000002, 2023-11-01T04:16:55.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52628482899999995, 2023-11-01T04:16:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:55.249 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 565 2023-11-01T04:16:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511279+0000", 2023-11-01T04:16:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511279+0000", 2023-11-01T04:16:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511282+0000", 2023-11-01T04:16:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5969999999999999e-06 2023-11-01T04:16:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511294+0000", 2023-11-01T04:16:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1776e-05 2023-11-01T04:16:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511326+0000", 2023-11-01T04:16:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1686000000000001e-05 2023-11-01T04:16:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511341+0000", 2023-11-01T04:16:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5475999999999999e-05 2023-11-01T04:16:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511529+0000", 2023-11-01T04:16:55.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018760700000000001 2023-11-01T04:16:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511739+0000", 2023-11-01T04:16:55.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00021064 2023-11-01T04:16:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.037564+0000", 2023-11-01T04:16:55.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52582504699999999 2023-11-01T04:16:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:566 3.2s0 3:42f00163:::smithi19039307-2:head [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2023-11-01T04:16:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:49.511373+0000", 2023-11-01T04:16:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.4986316210000004, 2023-11-01T04:16:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52738665100000004, 2023-11-01T04:16:55.260 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:55.261 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 566 2023-11-01T04:16:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511373+0000", 2023-11-01T04:16:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511373+0000", 2023-11-01T04:16:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511374+0000", 2023-11-01T04:16:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3170000000000001e-06 2023-11-01T04:16:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511381+0000", 2023-11-01T04:16:55.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.776e-06 2023-11-01T04:16:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511412+0000", 2023-11-01T04:16:55.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1093000000000002e-05 2023-11-01T04:16:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511431+0000", 2023-11-01T04:16:55.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8799000000000001e-05 2023-11-01T04:16:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.512308+0000", 2023-11-01T04:16:55.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00087712700000000001 2023-11-01T04:16:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.512476+0000", 2023-11-01T04:16:55.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016766700000000001 2023-11-01T04:16:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.038760+0000", 2023-11-01T04:16:55.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52628387200000004 2023-11-01T04:16:55.271 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.271 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.271 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:567 3.2s0 3:42f00163:::smithi19039307-2:head [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2023-11-01T04:16:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:49.511529+0000", 2023-11-01T04:16:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.4984757169999998, 2023-11-01T04:16:55.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72495039999999999, 2023-11-01T04:16:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:16:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:55.273 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 567 2023-11-01T04:16:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511529+0000", 2023-11-01T04:16:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511529+0000", 2023-11-01T04:16:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511530+0000", 2023-11-01T04:16:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2759999999999999e-06 2023-11-01T04:16:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511537+0000", 2023-11-01T04:16:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2899999999999997e-06 2023-11-01T04:16:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511557+0000", 2023-11-01T04:16:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9443e-05 2023-11-01T04:16:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511577+0000", 2023-11-01T04:16:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9673e-05 2023-11-01T04:16:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.512998+0000", 2023-11-01T04:16:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014214360000000001 2023-11-01T04:16:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.513127+0000", 2023-11-01T04:16:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012906500000000001 2023-11-01T04:16:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.236479+0000", 2023-11-01T04:16:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72335221699999996 2023-11-01T04:16:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.283 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.284 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpReadReply(3.2s0 69/31 ECSubReadReply(tid=322, attrs_read=0))", 2023-11-01T04:16:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:49.514831+0000", 2023-11-01T04:16:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.4951732929999997, 2023-11-01T04:16:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72027060200000004, 2023-11-01T04:16:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.285 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.514831+0000", 2023-11-01T04:16:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.514831+0000", 2023-11-01T04:16:55.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.514831+0000", 2023-11-01T04:16:55.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.514831+0000", 2023-11-01T04:16:55.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.614593+0000", 2023-11-01T04:16:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.099762016999999995 2023-11-01T04:16:55.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.714692+0000", 2023-11-01T04:16:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10009886599999999 2023-11-01T04:16:55.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.235030+0000", 2023-11-01T04:16:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52033755299999995 2023-11-01T04:16:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.235102+0000", 2023-11-01T04:16:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2166e-05 2023-11-01T04:16:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.293 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:571 3.2s0 3:4a13469b:::smithi19039307-24:head [setxattr _header (57) in=7b] snapc 22=[22,1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2023-11-01T04:16:55.294 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:49.541372+0000", 2023-11-01T04:16:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.4686329459999996, 2023-11-01T04:16:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98667972400000004, 2023-11-01T04:16:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:16:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:55.295 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:16:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:16:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 571 2023-11-01T04:16:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.296 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:16:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:16:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541372+0000", 2023-11-01T04:16:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:16:55.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541372+0000", 2023-11-01T04:16:55.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:16:55.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:16:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541373+0000", 2023-11-01T04:16:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8300000000000001e-06 2023-11-01T04:16:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:16:55.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541386+0000", 2023-11-01T04:16:55.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2146e-05 2023-11-01T04:16:55.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:16:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541410+0000", 2023-11-01T04:16:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4672999999999999e-05 2023-11-01T04:16:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541427+0000", 2023-11-01T04:16:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6796e-05 2023-11-01T04:16:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:16:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.236577+0000", 2023-11-01T04:16:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.69514989800000004 2023-11-01T04:16:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:16:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.236756+0000", 2023-11-01T04:16:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000179211 2023-11-01T04:16:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.237530+0000", 2023-11-01T04:16:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077358099999999996 2023-11-01T04:16:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.238841+0000", 2023-11-01T04:16:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001311682 2023-11-01T04:16:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:16:55.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.527800+0000", 2023-11-01T04:16:55.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28895858200000002 2023-11-01T04:16:55.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:16:55.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:16:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:16:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.528051+0000", 2023-11-01T04:16:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000251325 2023-11-01T04:16:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.308 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:16:55.309 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:16:55.309 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:16:55.312 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-01T04:16:55.313 INFO:tasks.rados.rados.0.smithi190.stdout:249: done (4 left) 2023-11-01T04:16:55.313 INFO:tasks.rados.rados.0.smithi190.stdout:250: done (3 left) 2023-11-01T04:16:55.313 INFO:tasks.rados.rados.0.smithi190.stdout:256: done (2 left) 2023-11-01T04:16:55.313 INFO:tasks.rados.rados.0.smithi190.stdout:257: rmattr oid 24 current snap is 23 2023-11-01T04:16:55.313 INFO:tasks.rados.rados.0.smithi190.stdout:257: done (2 left) 2023-11-01T04:16:55.314 INFO:tasks.rados.rados.0.smithi190.stdout:258: snap_create 2023-11-01T04:16:55.314 INFO:tasks.rados.rados.0.smithi190.stdout:253: finishing write tid 1 to smithi19039307-23 2023-11-01T04:16:55.314 INFO:tasks.rados.rados.0.smithi190.stdout:253: finishing write tid 2 to smithi19039307-23 2023-11-01T04:16:55.314 INFO:tasks.rados.rados.0.smithi190.stdout:253: finishing write tid 3 to smithi19039307-23 2023-11-01T04:16:55.314 INFO:tasks.rados.rados.0.smithi190.stdout:254: finishing rollback tid 0 to smithi19039307-26 2023-11-01T04:16:55.315 INFO:tasks.rados.rados.0.smithi190.stdout:254: finishing rollback tid 1 to smithi19039307-26 2023-11-01T04:16:55.315 INFO:tasks.rados.rados.0.smithi190.stdout:254: finishing rollback tid 2 to smithi19039307-26 2023-11-01T04:16:55.315 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 26 v 34 (ObjNum 25 snap 0 seq_num 25) dirty exists 2023-11-01T04:16:55.316 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:55.307+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:55.406 INFO:tasks.rados.rados.0.smithi190.stdout:253: finishing write tid 4 to smithi19039307-23 2023-11-01T04:16:55.411 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:55.412+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:55.450 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:55.450 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2023-11-01T04:16:55.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.451 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.ds5 74/68 ECSubRead(tid=16, to_read={3:ba00e0a8:::smithi19039307-23:head=0,4096,0}, subchunks={3:ba00e0a8:::smithi19039307-23:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:55.451 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:55.407593+0000", 2023-11-01T04:16:55.451 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.043386985000000003, 2023-11-01T04:16:55.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.043452119999999997, 2023-11-01T04:16:55.451 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.452 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:55.452 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.452 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.452 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.407593+0000", 2023-11-01T04:16:55.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.407593+0000", 2023-11-01T04:16:55.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.407596+0000", 2023-11-01T04:16:55.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9830000000000001e-06 2023-11-01T04:16:55.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.407623+0000", 2023-11-01T04:16:55.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6597e-05 2023-11-01T04:16:55.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.407663+0000", 2023-11-01T04:16:55.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0831999999999998e-05 2023-11-01T04:16:55.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.407684+0000", 2023-11-01T04:16:55.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0769999999999999e-05 2023-11-01T04:16:55.457 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.457 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.457 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.458 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.458 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2023-11-01T04:16:55.458 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 1 2023-11-01T04:16:55.458 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:55.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-01T04:16:55.512 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:55.510+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:55.581 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:16:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2023-11-01T04:16:55.582 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2023-11-01T04:16:55.582 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:55.595 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-01T04:16:55.604 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-01T04:16:55.608 INFO:tasks.rados.rados.0.smithi190.stdout:253: finishing write tid 5 to smithi19039307-23 2023-11-01T04:16:55.608 INFO:tasks.rados.rados.0.smithi190.stdout:253: oid 23 updating version 0 to 85 2023-11-01T04:16:55.608 INFO:tasks.rados.rados.0.smithi190.stdout:253: oid 23 updating version 85 to 86 2023-11-01T04:16:55.609 INFO:tasks.rados.rados.0.smithi190.stdout:253: oid 23 version 86 is already newer than 83 2023-11-01T04:16:55.609 INFO:tasks.rados.rados.0.smithi190.stdout:253: oid 23 version 86 is already newer than 84 2023-11-01T04:16:55.609 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 23 v 86 (ObjNum 104 snap 23 seq_num 104) dirty exists 2023-11-01T04:16:55.609 INFO:tasks.rados.rados.0.smithi190.stdout:253: left oid 23 (ObjNum 104 snap 23 seq_num 104) 2023-11-01T04:16:55.613 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:16:55.609+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:16:55.714 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:55.711+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:55.720 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:16:55.721 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2023-11-01T04:16:55.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2023-11-01T04:16:55.721 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2023-11-01T04:16:55.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.721 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 36/31 ECSubWrite(tid=26, reqid=client.4559.0:142, at_version=36'13, trim_to=0'0, roll_forward_to=35'12))", 2023-11-01T04:16:55.721 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:15.204322+0000", 2023-11-01T04:16:55.722 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 40.513220292, 2023-11-01T04:16:55.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102807592, 2023-11-01T04:16:55.722 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.722 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.722 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.204322+0000", 2023-11-01T04:16:55.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.204322+0000", 2023-11-01T04:16:55.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.204324+0000", 2023-11-01T04:16:55.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9400000000000001e-06 2023-11-01T04:16:55.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.204568+0000", 2023-11-01T04:16:55.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00024425800000000002 2023-11-01T04:16:55.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.204598+0000", 2023-11-01T04:16:55.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9845e-05 2023-11-01T04:16:55.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.204614+0000", 2023-11-01T04:16:55.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5919000000000001e-05 2023-11-01T04:16:55.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.304866+0000", 2023-11-01T04:16:55.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100252465 2023-11-01T04:16:55.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.304891+0000", 2023-11-01T04:16:55.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4068999999999999e-05 2023-11-01T04:16:55.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.307052+0000", 2023-11-01T04:16:55.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021609989999999998 2023-11-01T04:16:55.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.307130+0000", 2023-11-01T04:16:55.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8096999999999994e-05 2023-11-01T04:16:55.732 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.732 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.732 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.732 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.3s2 36/31 ECSubRead(tid=196, to_read={3:c87735c0:::smithi19039307-50:head=0,4096,0}, subchunks={3:c87735c0:::smithi19039307-50:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:16:55.733 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:15.821807+0000", 2023-11-01T04:16:55.733 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 39.895734836999999, 2023-11-01T04:16:55.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20089694399999999, 2023-11-01T04:16:55.733 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.733 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.733 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.821807+0000", 2023-11-01T04:16:55.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.821807+0000", 2023-11-01T04:16:55.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.821809+0000", 2023-11-01T04:16:55.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.296e-06 2023-11-01T04:16:55.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.821825+0000", 2023-11-01T04:16:55.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6463000000000001e-05 2023-11-01T04:16:55.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.821853+0000", 2023-11-01T04:16:55.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7512e-05 2023-11-01T04:16:55.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.921962+0000", 2023-11-01T04:16:55.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100109578 2023-11-01T04:16:55.739 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.739 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.739 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.739 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.022264+0000", 2023-11-01T04:16:55.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100301818 2023-11-01T04:16:55.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.022704+0000", 2023-11-01T04:16:55.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00044027699999999999 2023-11-01T04:16:55.740 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.741 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.741 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.741 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.741 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 37/31 ECSubWrite(tid=22, reqid=client.4559.0:187, at_version=37'15, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:16:55.742 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.969636+0000", 2023-11-01T04:16:55.742 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.747906245999999, 2023-11-01T04:16:55.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19731178799999999, 2023-11-01T04:16:55.742 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.742 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.742 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969636+0000", 2023-11-01T04:16:55.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969636+0000", 2023-11-01T04:16:55.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969638+0000", 2023-11-01T04:16:55.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9199999999999998e-06 2023-11-01T04:16:55.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969695+0000", 2023-11-01T04:16:55.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7497999999999997e-05 2023-11-01T04:16:55.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.747 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969721+0000", 2023-11-01T04:16:55.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5024999999999998e-05 2023-11-01T04:16:55.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969736+0000", 2023-11-01T04:16:55.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5383000000000002e-05 2023-11-01T04:16:55.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.165457+0000", 2023-11-01T04:16:55.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19572152500000001 2023-11-01T04:16:55.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.165472+0000", 2023-11-01T04:16:55.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4713e-05 2023-11-01T04:16:55.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.751 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.751 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.166866+0000", 2023-11-01T04:16:55.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013936039999999999 2023-11-01T04:16:55.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.752 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.166948+0000", 2023-11-01T04:16:55.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2120000000000007e-05 2023-11-01T04:16:55.752 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.752 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.753 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.753 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.fs6 37/31 ECSubWrite(tid=8, reqid=client.4559.0:188, at_version=37'4, trim_to=0'0, roll_forward_to=35'2))", 2023-11-01T04:16:55.753 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.169234+0000", 2023-11-01T04:16:55.753 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.548308740000003, 2023-11-01T04:16:55.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10309233800000001, 2023-11-01T04:16:55.754 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.754 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.754 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.754 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.754 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.169234+0000", 2023-11-01T04:16:55.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.169234+0000", 2023-11-01T04:16:55.756 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.756 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.756 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.756 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.169234+0000", 2023-11-01T04:16:55.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0699999999999998e-07 2023-11-01T04:16:55.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.169374+0000", 2023-11-01T04:16:55.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000139665 2023-11-01T04:16:55.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.169403+0000", 2023-11-01T04:16:55.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8886000000000001e-05 2023-11-01T04:16:55.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.759 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.269510+0000", 2023-11-01T04:16:55.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100107338 2023-11-01T04:16:55.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.269778+0000", 2023-11-01T04:16:55.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00026818700000000003 2023-11-01T04:16:55.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.269837+0000", 2023-11-01T04:16:55.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8913999999999999e-05 2023-11-01T04:16:55.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.272242+0000", 2023-11-01T04:16:55.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002404851 2023-11-01T04:16:55.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.272326+0000", 2023-11-01T04:16:55.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4090000000000003e-05 2023-11-01T04:16:55.763 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.763 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.763 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.764 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.3s2 37/31 ECSubWrite(tid=208, reqid=client.4559.0:212, at_version=37'9, trim_to=0'0, roll_forward_to=36'7))", 2023-11-01T04:16:55.764 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.580549+0000", 2023-11-01T04:16:55.764 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.136993648000001, 2023-11-01T04:16:55.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103548872, 2023-11-01T04:16:55.764 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.765 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.765 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.580549+0000", 2023-11-01T04:16:55.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.580549+0000", 2023-11-01T04:16:55.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.580551+0000", 2023-11-01T04:16:55.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8929999999999999e-06 2023-11-01T04:16:55.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.580689+0000", 2023-11-01T04:16:55.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000138498 2023-11-01T04:16:55.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.580739+0000", 2023-11-01T04:16:55.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0049000000000001e-05 2023-11-01T04:16:55.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.680884+0000", 2023-11-01T04:16:55.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100145 2023-11-01T04:16:55.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.681059+0000", 2023-11-01T04:16:55.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000175181 2023-11-01T04:16:55.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.681084+0000", 2023-11-01T04:16:55.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5151999999999999e-05 2023-11-01T04:16:55.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.684027+0000", 2023-11-01T04:16:55.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002942475 2023-11-01T04:16:55.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.684098+0000", 2023-11-01T04:16:55.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0623999999999993e-05 2023-11-01T04:16:55.774 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.774 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.774 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.775 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 41/31 ECSubWrite(tid=56, reqid=client.4559.0:267, at_version=41'22, trim_to=0'0, roll_forward_to=37'19))", 2023-11-01T04:16:55.775 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:21.120537+0000", 2023-11-01T04:16:55.775 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.597005084000003, 2023-11-01T04:16:55.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102796897, 2023-11-01T04:16:55.775 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.776 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.776 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.120537+0000", 2023-11-01T04:16:55.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.120537+0000", 2023-11-01T04:16:55.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.120538+0000", 2023-11-01T04:16:55.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2270000000000001e-06 2023-11-01T04:16:55.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.120904+0000", 2023-11-01T04:16:55.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00036540100000000001 2023-11-01T04:16:55.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.121052+0000", 2023-11-01T04:16:55.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014824200000000001 2023-11-01T04:16:55.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.121073+0000", 2023-11-01T04:16:55.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0519000000000001e-05 2023-11-01T04:16:55.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.782 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.121204+0000", 2023-11-01T04:16:55.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013181 2023-11-01T04:16:55.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.121221+0000", 2023-11-01T04:16:55.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6449000000000001e-05 2023-11-01T04:16:55.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.223266+0000", 2023-11-01T04:16:55.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102045108 2023-11-01T04:16:55.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.223334+0000", 2023-11-01T04:16:55.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8140999999999997e-05 2023-11-01T04:16:55.785 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.785 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.785 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.786 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 41/31 ECSubWrite(tid=58, reqid=osd.12.0:58, at_version=41'25, trim_to=0'0, roll_forward_to=37'19))", 2023-11-01T04:16:55.786 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:21.122902+0000", 2023-11-01T04:16:55.786 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.594640318000003, 2023-11-01T04:16:55.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105779886, 2023-11-01T04:16:55.786 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.786 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.787 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.122902+0000", 2023-11-01T04:16:55.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.122902+0000", 2023-11-01T04:16:55.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.122903+0000", 2023-11-01T04:16:55.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3370000000000001e-06 2023-11-01T04:16:55.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.790 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.790 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.122949+0000", 2023-11-01T04:16:55.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5649000000000003e-05 2023-11-01T04:16:55.790 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.790 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.790 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.122981+0000", 2023-11-01T04:16:55.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2502000000000002e-05 2023-11-01T04:16:55.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.223072+0000", 2023-11-01T04:16:55.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100090422 2023-11-01T04:16:55.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.223448+0000", 2023-11-01T04:16:55.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00037613399999999997 2023-11-01T04:16:55.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.223471+0000", 2023-11-01T04:16:55.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3088999999999999e-05 2023-11-01T04:16:55.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.228640+0000", 2023-11-01T04:16:55.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051686800000000001 2023-11-01T04:16:55.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.228682+0000", 2023-11-01T04:16:55.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2073000000000002e-05 2023-11-01T04:16:55.796 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.796 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.796 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.796 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.3s2 47/31 ECSubWrite(tid=244, reqid=client.4559.0:359, at_version=47'14, trim_to=0'0, roll_forward_to=41'11))", 2023-11-01T04:16:55.797 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:27.511910+0000", 2023-11-01T04:16:55.797 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.205632681000001, 2023-11-01T04:16:55.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102941901, 2023-11-01T04:16:55.797 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.797 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.797 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.511910+0000", 2023-11-01T04:16:55.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.511910+0000", 2023-11-01T04:16:55.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.511919+0000", 2023-11-01T04:16:55.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6199999999999994e-06 2023-11-01T04:16:55.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.512087+0000", 2023-11-01T04:16:55.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016782200000000001 2023-11-01T04:16:55.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.512119+0000", 2023-11-01T04:16:55.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1761999999999998e-05 2023-11-01T04:16:55.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.512141+0000", 2023-11-01T04:16:55.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2643999999999999e-05 2023-11-01T04:16:55.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.612358+0000", 2023-11-01T04:16:55.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100216728 2023-11-01T04:16:55.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.612374+0000", 2023-11-01T04:16:55.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5773e-05 2023-11-01T04:16:55.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.614749+0000", 2023-11-01T04:16:55.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023747579999999998 2023-11-01T04:16:55.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.614852+0000", 2023-11-01T04:16:55.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010279400000000001 2023-11-01T04:16:55.807 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.807 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.807 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.808 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 53/31 ECSubWrite(tid=82, reqid=client.4559.0:405, at_version=53'37, trim_to=0'0, roll_forward_to=50'35))", 2023-11-01T04:16:55.808 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:33.313117+0000", 2023-11-01T04:16:55.808 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.404425014000001, 2023-11-01T04:16:55.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104799267, 2023-11-01T04:16:55.808 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.809 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.809 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313117+0000", 2023-11-01T04:16:55.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313117+0000", 2023-11-01T04:16:55.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313118+0000", 2023-11-01T04:16:55.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9999999999999998e-07 2023-11-01T04:16:55.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313430+0000", 2023-11-01T04:16:55.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00031231199999999997 2023-11-01T04:16:55.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313460+0000", 2023-11-01T04:16:55.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9879999999999999e-05 2023-11-01T04:16:55.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313477+0000", 2023-11-01T04:16:55.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7186000000000001e-05 2023-11-01T04:16:55.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313619+0000", 2023-11-01T04:16:55.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014189699999999999 2023-11-01T04:16:55.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313648+0000", 2023-11-01T04:16:55.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8812000000000001e-05 2023-11-01T04:16:55.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.417809+0000", 2023-11-01T04:16:55.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104160975 2023-11-01T04:16:55.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.417917+0000", 2023-11-01T04:16:55.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107705 2023-11-01T04:16:55.822 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.822 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.822 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.823 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 57/31 ECSubWrite(tid=103, reqid=osd.12.0:103, at_version=57'54, trim_to=0'0, roll_forward_to=55'52))", 2023-11-01T04:16:55.823 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:37.235471+0000", 2023-11-01T04:16:55.823 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 18.482071045000001, 2023-11-01T04:16:55.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10343498700000001, 2023-11-01T04:16:55.824 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.824 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.824 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.235471+0000", 2023-11-01T04:16:55.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.235471+0000", 2023-11-01T04:16:55.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.235472+0000", 2023-11-01T04:16:55.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5700000000000002e-07 2023-11-01T04:16:55.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.235526+0000", 2023-11-01T04:16:55.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3761999999999998e-05 2023-11-01T04:16:55.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.235558+0000", 2023-11-01T04:16:55.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2793000000000003e-05 2023-11-01T04:16:55.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.235586+0000", 2023-11-01T04:16:55.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7506000000000001e-05 2023-11-01T04:16:55.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.237808+0000", 2023-11-01T04:16:55.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022221659999999998 2023-11-01T04:16:55.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.237833+0000", 2023-11-01T04:16:55.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4709e-05 2023-11-01T04:16:55.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.338851+0000", 2023-11-01T04:16:55.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101018492 2023-11-01T04:16:55.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.338906+0000", 2023-11-01T04:16:55.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5002e-05 2023-11-01T04:16:55.833 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.833 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.833 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.834 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 57/31 ECSubWrite(tid=104, reqid=osd.12.0:104, at_version=57'56, trim_to=0'0, roll_forward_to=55'52))", 2023-11-01T04:16:55.834 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:37.237714+0000", 2023-11-01T04:16:55.834 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 18.479828722000001, 2023-11-01T04:16:55.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104091738, 2023-11-01T04:16:55.835 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.835 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.835 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.237714+0000", 2023-11-01T04:16:55.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.237714+0000", 2023-11-01T04:16:55.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.237715+0000", 2023-11-01T04:16:55.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.223e-06 2023-11-01T04:16:55.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.237765+0000", 2023-11-01T04:16:55.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9771999999999997e-05 2023-11-01T04:16:55.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.237798+0000", 2023-11-01T04:16:55.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3732999999999997e-05 2023-11-01T04:16:55.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.237822+0000", 2023-11-01T04:16:55.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3181999999999999e-05 2023-11-01T04:16:55.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.338057+0000", 2023-11-01T04:16:55.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10023538999999999 2023-11-01T04:16:55.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.338079+0000", 2023-11-01T04:16:55.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1820000000000001e-05 2023-11-01T04:16:55.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.341719+0000", 2023-11-01T04:16:55.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003640407 2023-11-01T04:16:55.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.341805+0000", 2023-11-01T04:16:55.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6211000000000005e-05 2023-11-01T04:16:55.846 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.846 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.847 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.847 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 67/31 ECSubWrite(tid=77, reqid=client.4559.0:537, at_version=67'58, trim_to=0'0, roll_forward_to=67'56))", 2023-11-01T04:16:55.847 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:47.705575+0000", 2023-11-01T04:16:55.847 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 8.0119672229999992, 2023-11-01T04:16:55.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.198004336, 2023-11-01T04:16:55.848 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.848 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.848 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.705575+0000", 2023-11-01T04:16:55.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.705575+0000", 2023-11-01T04:16:55.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.705576+0000", 2023-11-01T04:16:55.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6199999999999996e-07 2023-11-01T04:16:55.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.706201+0000", 2023-11-01T04:16:55.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00062493599999999998 2023-11-01T04:16:55.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.706232+0000", 2023-11-01T04:16:55.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0766000000000003e-05 2023-11-01T04:16:55.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.706256+0000", 2023-11-01T04:16:55.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4358999999999998e-05 2023-11-01T04:16:55.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.802580+0000", 2023-11-01T04:16:55.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.096323874000000004 2023-11-01T04:16:55.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.802593+0000", 2023-11-01T04:16:55.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2634e-05 2023-11-01T04:16:55.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.903487+0000", 2023-11-01T04:16:55.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10089468 2023-11-01T04:16:55.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.903579+0000", 2023-11-01T04:16:55.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2225000000000006e-05 2023-11-01T04:16:55.858 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.858 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.858 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.859 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 68/31 ECSubWrite(tid=79, reqid=osd.11.0:79, at_version=68'62, trim_to=0'0, roll_forward_to=67'56))", 2023-11-01T04:16:55.859 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:47.907450+0000", 2023-11-01T04:16:55.859 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 7.8100923890000002, 2023-11-01T04:16:55.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10289667700000001, 2023-11-01T04:16:55.859 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.859 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.860 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.907450+0000", 2023-11-01T04:16:55.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.907450+0000", 2023-11-01T04:16:55.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.907451+0000", 2023-11-01T04:16:55.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2699999999999999e-06 2023-11-01T04:16:55.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.907485+0000", 2023-11-01T04:16:55.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4298999999999998e-05 2023-11-01T04:16:55.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.907508+0000", 2023-11-01T04:16:55.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2310000000000002e-05 2023-11-01T04:16:55.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.007607+0000", 2023-11-01T04:16:55.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10009870899999999 2023-11-01T04:16:55.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.007786+0000", 2023-11-01T04:16:55.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000179834 2023-11-01T04:16:55.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.007814+0000", 2023-11-01T04:16:55.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7350000000000001e-05 2023-11-01T04:16:55.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.010290+0000", 2023-11-01T04:16:55.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024767690000000002 2023-11-01T04:16:55.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.010347+0000", 2023-11-01T04:16:55.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6135999999999999e-05 2023-11-01T04:16:55.869 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.869 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.869 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.870 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s4 69/31 ECSubWrite(tid=85, reqid=client.4559.0:555, at_version=69'46, trim_to=0'0, roll_forward_to=68'44))", 2023-11-01T04:16:55.870 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:48.790183+0000", 2023-11-01T04:16:55.870 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.9273588329999995, 2023-11-01T04:16:55.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20311115800000001, 2023-11-01T04:16:55.871 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.871 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.871 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.790183+0000", 2023-11-01T04:16:55.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.790183+0000", 2023-11-01T04:16:55.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.790185+0000", 2023-11-01T04:16:55.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4729999999999999e-06 2023-11-01T04:16:55.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.790243+0000", 2023-11-01T04:16:55.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7843000000000002e-05 2023-11-01T04:16:55.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.790282+0000", 2023-11-01T04:16:55.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9255999999999998e-05 2023-11-01T04:16:55.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.890377+0000", 2023-11-01T04:16:55.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100094456 2023-11-01T04:16:55.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.990579+0000", 2023-11-01T04:16:55.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100202728 2023-11-01T04:16:55.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.990596+0000", 2023-11-01T04:16:55.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6906999999999999e-05 2023-11-01T04:16:55.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.993193+0000", 2023-11-01T04:16:55.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025967630000000002 2023-11-01T04:16:55.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.993295+0000", 2023-11-01T04:16:55.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010173199999999999 2023-11-01T04:16:55.880 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.880 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.881 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.881 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 69/31 ECSubWrite(tid=82, reqid=client.4559.0:561, at_version=69'64, trim_to=0'0, roll_forward_to=68'62))", 2023-11-01T04:16:55.881 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:49.711973+0000", 2023-11-01T04:16:55.882 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.0055689040000004, 2023-11-01T04:16:55.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105150903, 2023-11-01T04:16:55.882 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.882 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.882 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.711973+0000", 2023-11-01T04:16:55.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.711973+0000", 2023-11-01T04:16:55.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.711975+0000", 2023-11-01T04:16:55.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9400000000000001e-06 2023-11-01T04:16:55.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712340+0000", 2023-11-01T04:16:55.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00036447999999999998 2023-11-01T04:16:55.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712372+0000", 2023-11-01T04:16:55.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2113000000000002e-05 2023-11-01T04:16:55.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712388+0000", 2023-11-01T04:16:55.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5835999999999999e-05 2023-11-01T04:16:55.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712536+0000", 2023-11-01T04:16:55.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014780800000000001 2023-11-01T04:16:55.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712563+0000", 2023-11-01T04:16:55.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6951999999999999e-05 2023-11-01T04:16:55.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.817062+0000", 2023-11-01T04:16:55.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104499095 2023-11-01T04:16:55.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.817124+0000", 2023-11-01T04:16:55.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2679000000000001e-05 2023-11-01T04:16:55.892 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.892 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.892 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.892 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 69/31 ECSubWrite(tid=83, reqid=client.4559.0:562, at_version=69'65, trim_to=0'0, roll_forward_to=68'62))", 2023-11-01T04:16:55.893 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:49.712424+0000", 2023-11-01T04:16:55.893 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.0051185020000002, 2023-11-01T04:16:55.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104741764, 2023-11-01T04:16:55.893 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.893 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.894 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712424+0000", 2023-11-01T04:16:55.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712424+0000", 2023-11-01T04:16:55.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712425+0000", 2023-11-01T04:16:55.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9299999999999997e-07 2023-11-01T04:16:55.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712599+0000", 2023-11-01T04:16:55.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017441700000000001 2023-11-01T04:16:55.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712615+0000", 2023-11-01T04:16:55.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6120000000000002e-05 2023-11-01T04:16:55.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712628+0000", 2023-11-01T04:16:55.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3223e-05 2023-11-01T04:16:55.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.713736+0000", 2023-11-01T04:16:55.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011081660000000001 2023-11-01T04:16:55.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.713764+0000", 2023-11-01T04:16:55.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7708999999999998e-05 2023-11-01T04:16:55.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.817134+0000", 2023-11-01T04:16:55.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103369669 2023-11-01T04:16:55.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.817166+0000", 2023-11-01T04:16:55.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1767000000000003e-05 2023-11-01T04:16:55.903 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.903 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.903 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.904 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 69/31 ECSubWrite(tid=84, reqid=client.4559.0:563, at_version=69'66, trim_to=0'0, roll_forward_to=68'62))", 2023-11-01T04:16:55.904 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:49.712651+0000", 2023-11-01T04:16:55.904 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.00489096, 2023-11-01T04:16:55.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10454166400000001, 2023-11-01T04:16:55.905 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.905 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.905 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712651+0000", 2023-11-01T04:16:55.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712651+0000", 2023-11-01T04:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712652+0000", 2023-11-01T04:16:55.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3300000000000002e-07 2023-11-01T04:16:55.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712671+0000", 2023-11-01T04:16:55.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9174e-05 2023-11-01T04:16:55.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712684+0000", 2023-11-01T04:16:55.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2723e-05 2023-11-01T04:16:55.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712705+0000", 2023-11-01T04:16:55.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1229e-05 2023-11-01T04:16:55.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.815270+0000", 2023-11-01T04:16:55.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102564498 2023-11-01T04:16:55.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.815286+0000", 2023-11-01T04:16:55.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5903e-05 2023-11-01T04:16:55.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.817168+0000", 2023-11-01T04:16:55.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001882512 2023-11-01T04:16:55.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.817193+0000", 2023-11-01T04:16:55.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4992000000000001e-05 2023-11-01T04:16:55.914 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.915 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.915 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.915 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 69/31 ECSubWrite(tid=85, reqid=client.4559.0:573, at_version=69'68, trim_to=0'0, roll_forward_to=69'63))", 2023-11-01T04:16:55.915 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:49.714753+0000", 2023-11-01T04:16:55.916 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.0027894660000003, 2023-11-01T04:16:55.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10370937600000001, 2023-11-01T04:16:55.916 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.916 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.916 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.714753+0000", 2023-11-01T04:16:55.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.918 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.714753+0000", 2023-11-01T04:16:55.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.714755+0000", 2023-11-01T04:16:55.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7e-06 2023-11-01T04:16:55.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.714801+0000", 2023-11-01T04:16:55.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6415999999999997e-05 2023-11-01T04:16:55.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.714833+0000", 2023-11-01T04:16:55.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1659999999999998e-05 2023-11-01T04:16:55.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.714849+0000", 2023-11-01T04:16:55.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6008999999999999e-05 2023-11-01T04:16:55.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.815789+0000", 2023-11-01T04:16:55.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100940838 2023-11-01T04:16:55.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.815810+0000", 2023-11-01T04:16:55.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0100000000000001e-05 2023-11-01T04:16:55.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.818396+0000", 2023-11-01T04:16:55.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002586428 2023-11-01T04:16:55.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.818462+0000", 2023-11-01T04:16:55.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6224999999999997e-05 2023-11-01T04:16:55.925 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.925 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.925 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.926 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s4 73/72 ECSubWrite(tid=107, reqid=client.4559.0:611, at_version=73'62, trim_to=0'0, roll_forward_to=0'0))", 2023-11-01T04:16:55.926 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:54.133863+0000", 2023-11-01T04:16:55.926 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.5836797250000001, 2023-11-01T04:16:55.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104046954, 2023-11-01T04:16:55.927 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.927 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.927 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.927 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.133863+0000", 2023-11-01T04:16:55.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.133863+0000", 2023-11-01T04:16:55.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.133865+0000", 2023-11-01T04:16:55.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0159999999999998e-06 2023-11-01T04:16:55.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.134252+0000", 2023-11-01T04:16:55.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00038782399999999999 2023-11-01T04:16:55.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.931 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.134299+0000", 2023-11-01T04:16:55.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6816e-05 2023-11-01T04:16:55.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.234403+0000", 2023-11-01T04:16:55.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10010385300000001 2023-11-01T04:16:55.932 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.933 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.933 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.234573+0000", 2023-11-01T04:16:55.933 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000169617 2023-11-01T04:16:55.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.934 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.234607+0000", 2023-11-01T04:16:55.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4233000000000002e-05 2023-11-01T04:16:55.934 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.935 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.237826+0000", 2023-11-01T04:16:55.935 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032187829999999998 2023-11-01T04:16:55.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.935 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.935 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.935 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.237910+0000", 2023-11-01T04:16:55.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3812000000000003e-05 2023-11-01T04:16:55.936 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.936 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.936 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.936 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.937 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 74/68 ECSubWrite(tid=12, reqid=client.4559.0:621, at_version=74'85, trim_to=0'0, roll_forward_to=71'82))", 2023-11-01T04:16:55.937 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:54.975327+0000", 2023-11-01T04:16:55.937 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.74221556399999999, 2023-11-01T04:16:55.937 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102883153, 2023-11-01T04:16:55.938 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:16:55.938 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:16:55.938 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:16:55.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:16:55.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.975327+0000", 2023-11-01T04:16:55.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.940 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:16:55.940 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.975327+0000", 2023-11-01T04:16:55.940 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:16:55.940 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.940 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.940 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:16:55.941 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.975329+0000", 2023-11-01T04:16:55.941 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9999999999999999e-06 2023-11-01T04:16:55.941 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.941 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.941 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:16:55.941 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.975632+0000", 2023-11-01T04:16:55.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00030351899999999998 2023-11-01T04:16:55.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.942 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.942 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:16:55.942 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.975664+0000", 2023-11-01T04:16:55.943 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1483000000000003e-05 2023-11-01T04:16:55.943 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.943 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:16:55.943 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.975687+0000", 2023-11-01T04:16:55.943 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2826000000000001e-05 2023-11-01T04:16:55.944 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.944 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.944 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:16:55.944 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.075957+0000", 2023-11-01T04:16:55.944 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100270727 2023-11-01T04:16:55.944 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.945 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.945 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:16:55.945 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.075977+0000", 2023-11-01T04:16:55.945 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0046000000000001e-05 2023-11-01T04:16:55.945 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.945 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.946 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:16:55.946 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.078107+0000", 2023-11-01T04:16:55.946 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021297479999999999 2023-11-01T04:16:55.946 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:16:55.946 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:16:55.947 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:16:55.947 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.078210+0000", 2023-11-01T04:16:55.947 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000102804 2023-11-01T04:16:55.947 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.947 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.948 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.948 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:16:55.948 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:16:55.948 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:16:55.949 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:55.950 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-11-01T04:16:55.951 INFO:tasks.rados.rados.0.smithi190.stdout:253: done (2 left) 2023-11-01T04:16:55.951 INFO:tasks.rados.rados.0.smithi190.stdout:254: done (1 left) 2023-11-01T04:16:55.951 INFO:tasks.rados.rados.0.smithi190.stdout:258: done (0 left) 2023-11-01T04:16:55.951 INFO:tasks.rados.rados.0.smithi190.stdout:259: copy_from oid 48 from oid 47 current snap is 24 2023-11-01T04:16:55.951 INFO:tasks.rados.rados.0.smithi190.stdout:260: read oid 34 snap -1 2023-11-01T04:16:55.952 INFO:tasks.rados.rados.0.smithi190.stdout:260: expect (ObjNum 33 snap 0 seq_num 33) 2023-11-01T04:16:55.952 INFO:tasks.rados.rados.0.smithi190.stdout:261: snap_remove snap 11 2023-11-01T04:16:55.952 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:16:55.946+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:16:55.952 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:55.953 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-01T04:16:55.953 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-01T04:16:55.954 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:55.954 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-01T04:16:55.955 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:55.955 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-01T04:16:55.955 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:55.956 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-01T04:16:55.956 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:55.956 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-01T04:16:55.957 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:55.957 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-01T04:16:55.958 INFO:teuthology.orchestra.run.smithi153.stdout:osd.7: {} 2023-11-01T04:16:55.958 INFO:teuthology.orchestra.run.smithi153.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-11-01T04:16:55.958 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:16:55.959 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-01T04:16:55.959 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:16:55.959 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-01T04:16:55.974 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:16:55.974 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-01T04:16:55.996 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:16:55.996 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-01T04:16:55.997 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.12 2023-11-01T04:16:56.050 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~0s 2023-11-01T04:16:56.051 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:56.048+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:56.119 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:16:56.119 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-01T04:16:56.127 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:56.127 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:56.143 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-01T04:16:56.146 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:16:56.146 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-01T04:16:56.151 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:56.146+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:56.169 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:16:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-01T04:16:56.175 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-01T04:16:56.176 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-01T04:16:56.253 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:56.251+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:56.264 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:56.265 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:56.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-01T04:16:56.354 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:56.354+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:56.445 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:56.445 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:56.455 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:56.456+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:56.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-01T04:16:56.557 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:56.554+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:56.582 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:56.583 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:56.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-01T04:16:56.658 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:56.656+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:56.722 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:56.722 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:56.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-01T04:16:56.759 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:56.758+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:56.786 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-11-01T04:16:56.817 INFO:tasks.rados.rados.0.smithi190.stdout:261: done (2 left) 2023-11-01T04:16:56.817 INFO:tasks.rados.rados.0.smithi190.stdout:262: rollback oid 40 current snap is 24 2023-11-01T04:16:56.818 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 40 to 15 2023-11-01T04:16:56.818 INFO:tasks.rados.rados.0.smithi190.stdout:259: finishing copy_from racing read to smithi19039307-48 2023-11-01T04:16:56.818 INFO:tasks.rados.rados.0.smithi190.stdout:259: finishing copy_from to smithi19039307-48 2023-11-01T04:16:56.818 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 48 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2023-11-01T04:16:56.818 INFO:tasks.rados.rados.0.smithi190.stdout:260: expect (ObjNum 33 snap 0 seq_num 33) 2023-11-01T04:16:56.859 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:56.856+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:56.865 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:56.865 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:56.866 INFO:tasks.rados.rados.0.smithi190.stdout:259: done (2 left) 2023-11-01T04:16:56.866 INFO:tasks.rados.rados.0.smithi190.stdout:260: done (1 left) 2023-11-01T04:16:56.866 INFO:tasks.rados.rados.0.smithi190.stdout:263: read oid 32 snap -1 2023-11-01T04:16:56.866 INFO:tasks.rados.rados.0.smithi190.stdout:263: expect (ObjNum 53 snap 1 seq_num 53) 2023-11-01T04:16:56.867 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 46 current snap is 24 2023-11-01T04:16:56.867 INFO:tasks.rados.rados.0.smithi190.stdout:264: seq_num 106 ranges {770048=622592,1392640=442368,1835008=524288,2359296=196608} 2023-11-01T04:16:56.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-01T04:16:56.889 INFO:tasks.rados.rados.0.smithi190.stdout:264: writing smithi19039307-46 from 770048 to 1392640 tid 1 2023-11-01T04:16:56.896 INFO:tasks.rados.rados.0.smithi190.stdout:264: writing smithi19039307-46 from 1392640 to 1835008 tid 2 2023-11-01T04:16:56.905 INFO:tasks.rados.rados.0.smithi190.stdout:264: writing smithi19039307-46 from 1835008 to 2359296 tid 3 2023-11-01T04:16:56.908 INFO:tasks.rados.rados.0.smithi190.stdout:264: writing smithi19039307-46 from 2359296 to 2555904 tid 4 2023-11-01T04:16:56.909 INFO:tasks.rados.rados.0.smithi190.stdout:265: setattr oid 10 current snap is 24 2023-11-01T04:16:56.913 INFO:tasks.rados.rados.0.smithi190.stdout:266: setattr oid 43 current snap is 24 2023-11-01T04:16:56.917 INFO:tasks.rados.rados.0.smithi190.stdout:267: snap_remove snap 22 2023-11-01T04:16:56.960 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:56.957+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:56.999 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:56.999 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:57.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-01T04:16:57.061 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:57.058+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:57.139 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:57.140 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:57.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-01T04:16:57.162 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:57.161+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:57.263 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:57.261+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:57.283 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:57.283 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:57.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-11-01T04:16:57.363 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:16:57.433 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:57.433 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:57.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-11-01T04:16:57.466 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:16:57.463+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:16:57.567 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:16:57.563+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:16:57.571 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:57.571 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:57.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-11-01T04:16:57.668 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:57.668+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:57.713 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:57.713 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:57.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-01T04:16:57.769 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:57.768+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:57.769 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:57.765+0000 7fc0b5e58640 -1 snap_mapper.get_next_objects_to_trimNew Clone-Objects were added to Snap 11 after trimming was started 2023-11-01T04:16:57.770 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:57.765+0000 7fc0b5e58640 -1 snap_mapper.0x7fc0b5e54750 2023-11-01T04:16:57.776 INFO:tasks.ceph.osd.7.smithi190.stderr:./src/osd/SnapMapper.cc: In function 'int SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector*)' thread 7fc0b5e58640 time 2023-11-01T04:16:57.771425+0000 2023-11-01T04:16:57.776 INFO:tasks.ceph.osd.7.smithi190.stderr:./src/osd/SnapMapper.cc: 650: ceph_abort_msg("Clone-Objects were added to a trimmed snap") 2023-11-01T04:16:57.776 INFO:tasks.ceph.osd.7.smithi190.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:57.776 INFO:tasks.ceph.osd.7.smithi190.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x563d97ece9dc] 2023-11-01T04:16:57.777 INFO:tasks.ceph.osd.7.smithi190.stderr: 2: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x563d981b2c9d] 2023-11-01T04:16:57.777 INFO:tasks.ceph.osd.7.smithi190.stderr: 3: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x563d9811d2c2] 2023-11-01T04:16:57.777 INFO:tasks.ceph.osd.7.smithi190.stderr: 4: ceph-osd(+0xd7e721) [0x563d97f78721] 2023-11-01T04:16:57.777 INFO:tasks.ceph.osd.7.smithi190.stderr: 5: ceph-osd(+0x1695cbb) [0x563d9888fcbb] 2023-11-01T04:16:57.777 INFO:tasks.ceph.osd.7.smithi190.stderr: 6: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x563d980ba288] 2023-11-01T04:16:57.778 INFO:tasks.ceph.osd.7.smithi190.stderr: 7: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x563d981f09df] 2023-11-01T04:16:57.778 INFO:tasks.ceph.osd.7.smithi190.stderr: 8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x563d9800ed23] 2023-11-01T04:16:57.778 INFO:tasks.ceph.osd.7.smithi190.stderr: 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x563d984c0053] 2023-11-01T04:16:57.778 INFO:tasks.ceph.osd.7.smithi190.stderr: 10: ceph-osd(+0x12c6584) [0x563d984c0584] 2023-11-01T04:16:57.778 INFO:tasks.ceph.osd.7.smithi190.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc0d14c4b43] 2023-11-01T04:16:57.779 INFO:tasks.ceph.osd.7.smithi190.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc0d1556a00] 2023-11-01T04:16:57.779 INFO:tasks.ceph.osd.7.smithi190.stderr:*** Caught signal (Aborted) ** 2023-11-01T04:16:57.779 INFO:tasks.ceph.osd.7.smithi190.stderr: in thread 7fc0b5e58640 thread_name:tp_osd_tp 2023-11-01T04:16:57.779 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:57.773+0000 7fc0b5e58640 -1 ./src/osd/SnapMapper.cc: In function 'int SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector*)' thread 7fc0b5e58640 time 2023-11-01T04:16:57.771425+0000 2023-11-01T04:16:57.779 INFO:tasks.ceph.osd.7.smithi190.stderr:./src/osd/SnapMapper.cc: 650: ceph_abort_msg("Clone-Objects were added to a trimmed snap") 2023-11-01T04:16:57.780 INFO:tasks.ceph.osd.7.smithi190.stderr: 2023-11-01T04:16:57.780 INFO:tasks.ceph.osd.7.smithi190.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:57.780 INFO:tasks.ceph.osd.7.smithi190.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x563d97ece9dc] 2023-11-01T04:16:57.780 INFO:tasks.ceph.osd.7.smithi190.stderr: 2: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x563d981b2c9d] 2023-11-01T04:16:57.780 INFO:tasks.ceph.osd.7.smithi190.stderr: 3: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x563d9811d2c2] 2023-11-01T04:16:57.781 INFO:tasks.ceph.osd.7.smithi190.stderr: 4: ceph-osd(+0xd7e721) [0x563d97f78721] 2023-11-01T04:16:57.781 INFO:tasks.ceph.osd.7.smithi190.stderr: 5: ceph-osd(+0x1695cbb) [0x563d9888fcbb] 2023-11-01T04:16:57.781 INFO:tasks.ceph.osd.7.smithi190.stderr: 6: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x563d980ba288] 2023-11-01T04:16:57.781 INFO:tasks.ceph.osd.7.smithi190.stderr: 7: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x563d981f09df] 2023-11-01T04:16:57.781 INFO:tasks.ceph.osd.7.smithi190.stderr: 8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x563d9800ed23] 2023-11-01T04:16:57.782 INFO:tasks.ceph.osd.7.smithi190.stderr: 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x563d984c0053] 2023-11-01T04:16:57.782 INFO:tasks.ceph.osd.7.smithi190.stderr: 10: ceph-osd(+0x12c6584) [0x563d984c0584] 2023-11-01T04:16:57.782 INFO:tasks.ceph.osd.7.smithi190.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc0d14c4b43] 2023-11-01T04:16:57.782 INFO:tasks.ceph.osd.7.smithi190.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc0d1556a00] 2023-11-01T04:16:57.782 INFO:tasks.ceph.osd.7.smithi190.stderr: 2023-11-01T04:16:57.783 INFO:tasks.ceph.osd.7.smithi190.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:57.783 INFO:tasks.ceph.osd.7.smithi190.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fc0d1472520] 2023-11-01T04:16:57.784 INFO:tasks.ceph.osd.7.smithi190.stderr: 2: pthread_kill() 2023-11-01T04:16:57.784 INFO:tasks.ceph.osd.7.smithi190.stderr: 3: raise() 2023-11-01T04:16:57.784 INFO:tasks.ceph.osd.7.smithi190.stderr: 4: abort() 2023-11-01T04:16:57.784 INFO:tasks.ceph.osd.7.smithi190.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x563d97ecea91] 2023-11-01T04:16:57.784 INFO:tasks.ceph.osd.7.smithi190.stderr: 6: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x563d981b2c9d] 2023-11-01T04:16:57.784 INFO:tasks.ceph.osd.7.smithi190.stderr: 7: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x563d9811d2c2] 2023-11-01T04:16:57.785 INFO:tasks.ceph.osd.7.smithi190.stderr: 8: ceph-osd(+0xd7e721) [0x563d97f78721] 2023-11-01T04:16:57.785 INFO:tasks.ceph.osd.7.smithi190.stderr: 9: ceph-osd(+0x1695cbb) [0x563d9888fcbb] 2023-11-01T04:16:57.785 INFO:tasks.ceph.osd.7.smithi190.stderr: 10: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x563d980ba288] 2023-11-01T04:16:57.785 INFO:tasks.ceph.osd.7.smithi190.stderr: 11: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x563d981f09df] 2023-11-01T04:16:57.785 INFO:tasks.ceph.osd.7.smithi190.stderr: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x563d9800ed23] 2023-11-01T04:16:57.786 INFO:tasks.ceph.osd.7.smithi190.stderr: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x563d984c0053] 2023-11-01T04:16:57.786 INFO:tasks.ceph.osd.7.smithi190.stderr: 14: ceph-osd(+0x12c6584) [0x563d984c0584] 2023-11-01T04:16:57.786 INFO:tasks.ceph.osd.7.smithi190.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc0d14c4b43] 2023-11-01T04:16:57.786 INFO:tasks.ceph.osd.7.smithi190.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc0d1556a00] 2023-11-01T04:16:57.786 INFO:tasks.ceph.osd.7.smithi190.stderr:2023-11-01T04:16:57.777+0000 7fc0b5e58640 -1 *** Caught signal (Aborted) ** 2023-11-01T04:16:57.787 INFO:tasks.ceph.osd.7.smithi190.stderr: in thread 7fc0b5e58640 thread_name:tp_osd_tp 2023-11-01T04:16:57.787 INFO:tasks.ceph.osd.7.smithi190.stderr: 2023-11-01T04:16:57.787 INFO:tasks.ceph.osd.7.smithi190.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:57.787 INFO:tasks.ceph.osd.7.smithi190.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fc0d1472520] 2023-11-01T04:16:57.788 INFO:tasks.ceph.osd.7.smithi190.stderr: 2: pthread_kill() 2023-11-01T04:16:57.788 INFO:tasks.ceph.osd.7.smithi190.stderr: 3: raise() 2023-11-01T04:16:57.788 INFO:tasks.ceph.osd.7.smithi190.stderr: 4: abort() 2023-11-01T04:16:57.788 INFO:tasks.ceph.osd.7.smithi190.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x563d97ecea91] 2023-11-01T04:16:57.788 INFO:tasks.ceph.osd.7.smithi190.stderr: 6: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x563d981b2c9d] 2023-11-01T04:16:57.789 INFO:tasks.ceph.osd.7.smithi190.stderr: 7: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x563d9811d2c2] 2023-11-01T04:16:57.789 INFO:tasks.ceph.osd.7.smithi190.stderr: 8: ceph-osd(+0xd7e721) [0x563d97f78721] 2023-11-01T04:16:57.789 INFO:tasks.ceph.osd.7.smithi190.stderr: 9: ceph-osd(+0x1695cbb) [0x563d9888fcbb] 2023-11-01T04:16:57.789 INFO:tasks.ceph.osd.7.smithi190.stderr: 10: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x563d980ba288] 2023-11-01T04:16:57.789 INFO:tasks.ceph.osd.7.smithi190.stderr: 11: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x563d981f09df] 2023-11-01T04:16:57.790 INFO:tasks.ceph.osd.7.smithi190.stderr: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x563d9800ed23] 2023-11-01T04:16:57.790 INFO:tasks.ceph.osd.7.smithi190.stderr: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x563d984c0053] 2023-11-01T04:16:57.790 INFO:tasks.ceph.osd.7.smithi190.stderr: 14: ceph-osd(+0x12c6584) [0x563d984c0584] 2023-11-01T04:16:57.790 INFO:tasks.ceph.osd.7.smithi190.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc0d14c4b43] 2023-11-01T04:16:57.791 INFO:tasks.ceph.osd.7.smithi190.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc0d1556a00] 2023-11-01T04:16:57.791 INFO:tasks.ceph.osd.7.smithi190.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-01T04:16:57.791 INFO:tasks.ceph.osd.7.smithi190.stderr: 2023-11-01T04:16:57.794 INFO:tasks.ceph.osd.7.smithi190.stderr: -5574> 2023-11-01T04:16:56.146+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:57.795 INFO:tasks.ceph.osd.7.smithi190.stderr: -5486> 2023-11-01T04:16:56.554+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:57.801 INFO:tasks.ceph.osd.7.smithi190.stderr: -3> 2023-11-01T04:16:57.765+0000 7fc0b5e58640 -1 snap_mapper.get_next_objects_to_trimNew Clone-Objects were added to Snap 11 after trimming was started 2023-11-01T04:16:57.801 INFO:tasks.ceph.osd.7.smithi190.stderr: -2> 2023-11-01T04:16:57.765+0000 7fc0b5e58640 -1 snap_mapper.0x7fc0b5e54750 2023-11-01T04:16:57.801 INFO:tasks.ceph.osd.7.smithi190.stderr: -1> 2023-11-01T04:16:57.773+0000 7fc0b5e58640 -1 ./src/osd/SnapMapper.cc: In function 'int SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector*)' thread 7fc0b5e58640 time 2023-11-01T04:16:57.771425+0000 2023-11-01T04:16:57.801 INFO:tasks.ceph.osd.7.smithi190.stderr:./src/osd/SnapMapper.cc: 650: ceph_abort_msg("Clone-Objects were added to a trimmed snap") 2023-11-01T04:16:57.802 INFO:tasks.ceph.osd.7.smithi190.stderr: 2023-11-01T04:16:57.802 INFO:tasks.ceph.osd.7.smithi190.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:57.802 INFO:tasks.ceph.osd.7.smithi190.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x563d97ece9dc] 2023-11-01T04:16:57.802 INFO:tasks.ceph.osd.7.smithi190.stderr: 2: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x563d981b2c9d] 2023-11-01T04:16:57.802 INFO:tasks.ceph.osd.7.smithi190.stderr: 3: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x563d9811d2c2] 2023-11-01T04:16:57.803 INFO:tasks.ceph.osd.7.smithi190.stderr: 4: ceph-osd(+0xd7e721) [0x563d97f78721] 2023-11-01T04:16:57.803 INFO:tasks.ceph.osd.7.smithi190.stderr: 5: ceph-osd(+0x1695cbb) [0x563d9888fcbb] 2023-11-01T04:16:57.803 INFO:tasks.ceph.osd.7.smithi190.stderr: 6: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x563d980ba288] 2023-11-01T04:16:57.803 INFO:tasks.ceph.osd.7.smithi190.stderr: 7: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x563d981f09df] 2023-11-01T04:16:57.803 INFO:tasks.ceph.osd.7.smithi190.stderr: 8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x563d9800ed23] 2023-11-01T04:16:57.804 INFO:tasks.ceph.osd.7.smithi190.stderr: 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x563d984c0053] 2023-11-01T04:16:57.804 INFO:tasks.ceph.osd.7.smithi190.stderr: 10: ceph-osd(+0x12c6584) [0x563d984c0584] 2023-11-01T04:16:57.804 INFO:tasks.ceph.osd.7.smithi190.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc0d14c4b43] 2023-11-01T04:16:57.804 INFO:tasks.ceph.osd.7.smithi190.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc0d1556a00] 2023-11-01T04:16:57.804 INFO:tasks.ceph.osd.7.smithi190.stderr: 2023-11-01T04:16:57.805 INFO:tasks.ceph.osd.7.smithi190.stderr: 0> 2023-11-01T04:16:57.777+0000 7fc0b5e58640 -1 *** Caught signal (Aborted) ** 2023-11-01T04:16:57.805 INFO:tasks.ceph.osd.7.smithi190.stderr: in thread 7fc0b5e58640 thread_name:tp_osd_tp 2023-11-01T04:16:57.805 INFO:tasks.ceph.osd.7.smithi190.stderr: 2023-11-01T04:16:57.805 INFO:tasks.ceph.osd.7.smithi190.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:57.805 INFO:tasks.ceph.osd.7.smithi190.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fc0d1472520] 2023-11-01T04:16:57.805 INFO:tasks.ceph.osd.7.smithi190.stderr: 2: pthread_kill() 2023-11-01T04:16:57.806 INFO:tasks.ceph.osd.7.smithi190.stderr: 3: raise() 2023-11-01T04:16:57.806 INFO:tasks.ceph.osd.7.smithi190.stderr: 4: abort() 2023-11-01T04:16:57.806 INFO:tasks.ceph.osd.7.smithi190.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x563d97ecea91] 2023-11-01T04:16:57.806 INFO:tasks.ceph.osd.7.smithi190.stderr: 6: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x563d981b2c9d] 2023-11-01T04:16:57.807 INFO:tasks.ceph.osd.7.smithi190.stderr: 7: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x563d9811d2c2] 2023-11-01T04:16:57.807 INFO:tasks.ceph.osd.7.smithi190.stderr: 8: ceph-osd(+0xd7e721) [0x563d97f78721] 2023-11-01T04:16:57.807 INFO:tasks.ceph.osd.7.smithi190.stderr: 9: ceph-osd(+0x1695cbb) [0x563d9888fcbb] 2023-11-01T04:16:57.807 INFO:tasks.ceph.osd.7.smithi190.stderr: 10: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x563d980ba288] 2023-11-01T04:16:57.807 INFO:tasks.ceph.osd.7.smithi190.stderr: 11: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x563d981f09df] 2023-11-01T04:16:57.808 INFO:tasks.ceph.osd.7.smithi190.stderr: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x563d9800ed23] 2023-11-01T04:16:57.808 INFO:tasks.ceph.osd.7.smithi190.stderr: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x563d984c0053] 2023-11-01T04:16:57.808 INFO:tasks.ceph.osd.7.smithi190.stderr: 14: ceph-osd(+0x12c6584) [0x563d984c0584] 2023-11-01T04:16:57.808 INFO:tasks.ceph.osd.7.smithi190.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc0d14c4b43] 2023-11-01T04:16:57.808 INFO:tasks.ceph.osd.7.smithi190.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc0d1556a00] 2023-11-01T04:16:57.809 INFO:tasks.ceph.osd.7.smithi190.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-01T04:16:57.809 INFO:tasks.ceph.osd.7.smithi190.stderr: 2023-11-01T04:16:57.809 INFO:tasks.ceph.osd.7.smithi190.stderr: -5589> 2023-11-01T04:16:56.146+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:57.810 INFO:tasks.ceph.osd.7.smithi190.stderr: -5501> 2023-11-01T04:16:56.554+0000 7fc0cd866640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35951) UID: 0 2023-11-01T04:16:57.814 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [5] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2023-11-01T04:16:57.814 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-11-01T04:16:57.814 INFO:tasks.thrashosds.thrasher:inject_pause on osd.12 2023-11-01T04:16:57.815 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-11-01T04:16:57.815 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-11-01T04:16:57.815 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2023-11-01T04:16:57.816 INFO:tasks.ceph.osd.7.smithi190.stderr: -18> 2023-11-01T04:16:57.765+0000 7fc0b5e58640 -1 snap_mapper.get_next_objects_to_trimNew Clone-Objects were added to Snap 11 after trimming was started 2023-11-01T04:16:57.816 INFO:tasks.ceph.osd.7.smithi190.stderr: -17> 2023-11-01T04:16:57.765+0000 7fc0b5e58640 -1 snap_mapper.0x7fc0b5e54750 2023-11-01T04:16:57.816 INFO:tasks.ceph.osd.7.smithi190.stderr: -16> 2023-11-01T04:16:57.773+0000 7fc0b5e58640 -1 ./src/osd/SnapMapper.cc: In function 'int SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector*)' thread 7fc0b5e58640 time 2023-11-01T04:16:57.771425+0000 2023-11-01T04:16:57.817 INFO:tasks.ceph.osd.7.smithi190.stderr:./src/osd/SnapMapper.cc: 650: ceph_abort_msg("Clone-Objects were added to a trimmed snap") 2023-11-01T04:16:57.817 INFO:tasks.ceph.osd.7.smithi190.stderr: 2023-11-01T04:16:57.817 INFO:tasks.ceph.osd.7.smithi190.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:57.817 INFO:tasks.ceph.osd.7.smithi190.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x563d97ece9dc] 2023-11-01T04:16:57.817 INFO:tasks.ceph.osd.7.smithi190.stderr: 2: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x563d981b2c9d] 2023-11-01T04:16:57.818 INFO:tasks.ceph.osd.7.smithi190.stderr: 3: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x563d9811d2c2] 2023-11-01T04:16:57.818 INFO:tasks.ceph.osd.7.smithi190.stderr: 4: ceph-osd(+0xd7e721) [0x563d97f78721] 2023-11-01T04:16:57.818 INFO:tasks.ceph.osd.7.smithi190.stderr: 5: ceph-osd(+0x1695cbb) [0x563d9888fcbb] 2023-11-01T04:16:57.818 INFO:tasks.ceph.osd.7.smithi190.stderr: 6: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x563d980ba288] 2023-11-01T04:16:57.818 INFO:tasks.ceph.osd.7.smithi190.stderr: 7: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x563d981f09df] 2023-11-01T04:16:57.819 INFO:tasks.ceph.osd.7.smithi190.stderr: 8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x563d9800ed23] 2023-11-01T04:16:57.819 INFO:tasks.ceph.osd.7.smithi190.stderr: 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x563d984c0053] 2023-11-01T04:16:57.819 INFO:tasks.ceph.osd.7.smithi190.stderr: 10: ceph-osd(+0x12c6584) [0x563d984c0584] 2023-11-01T04:16:57.819 INFO:tasks.ceph.osd.7.smithi190.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc0d14c4b43] 2023-11-01T04:16:57.819 INFO:tasks.ceph.osd.7.smithi190.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc0d1556a00] 2023-11-01T04:16:57.819 INFO:tasks.ceph.osd.7.smithi190.stderr: 2023-11-01T04:16:57.820 INFO:tasks.ceph.osd.7.smithi190.stderr: -15> 2023-11-01T04:16:57.777+0000 7fc0b5e58640 -1 *** Caught signal (Aborted) ** 2023-11-01T04:16:57.820 INFO:tasks.ceph.osd.7.smithi190.stderr: in thread 7fc0b5e58640 thread_name:tp_osd_tp 2023-11-01T04:16:57.820 INFO:tasks.ceph.osd.7.smithi190.stderr: 2023-11-01T04:16:57.820 INFO:tasks.ceph.osd.7.smithi190.stderr: ceph version 18.0.0-6776-g420cd129 (420cd129397301b1521ceb74e1bdbd9b855d2189) reef (dev) 2023-11-01T04:16:57.820 INFO:tasks.ceph.osd.7.smithi190.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fc0d1472520] 2023-11-01T04:16:57.821 INFO:tasks.ceph.osd.7.smithi190.stderr: 2: pthread_kill() 2023-11-01T04:16:57.821 INFO:tasks.ceph.osd.7.smithi190.stderr: 3: raise() 2023-11-01T04:16:57.821 INFO:tasks.ceph.osd.7.smithi190.stderr: 4: abort() 2023-11-01T04:16:57.821 INFO:tasks.ceph.osd.7.smithi190.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x563d97ecea91] 2023-11-01T04:16:57.821 INFO:tasks.ceph.osd.7.smithi190.stderr: 6: (SnapMapper::get_next_objects_to_trim(snapid_t, unsigned int, std::vector >*)+0x4dd) [0x563d981b2c9d] 2023-11-01T04:16:57.822 INFO:tasks.ceph.osd.7.smithi190.stderr: 7: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork const&)+0x302) [0x563d9811d2c2] 2023-11-01T04:16:57.822 INFO:tasks.ceph.osd.7.smithi190.stderr: 8: ceph-osd(+0xd7e721) [0x563d97f78721] 2023-11-01T04:16:57.822 INFO:tasks.ceph.osd.7.smithi190.stderr: 9: ceph-osd(+0x1695cbb) [0x563d9888fcbb] 2023-11-01T04:16:57.822 INFO:tasks.ceph.osd.7.smithi190.stderr: 10: (PrimaryLogPG::snap_trimmer(unsigned int)+0xc8) [0x563d980ba288] 2023-11-01T04:16:57.822 INFO:tasks.ceph.osd.7.smithi190.stderr: 11: (ceph::osd::scheduler::PGSnapTrim::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x1f) [0x563d981f09df] 2023-11-01T04:16:57.823 INFO:tasks.ceph.osd.7.smithi190.stderr: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x563d9800ed23] 2023-11-01T04:16:57.823 INFO:tasks.ceph.osd.7.smithi190.stderr: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x233) [0x563d984c0053] 2023-11-01T04:16:57.823 INFO:tasks.ceph.osd.7.smithi190.stderr: 14: ceph-osd(+0x12c6584) [0x563d984c0584] 2023-11-01T04:16:57.823 INFO:tasks.ceph.osd.7.smithi190.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fc0d14c4b43] 2023-11-01T04:16:57.823 INFO:tasks.ceph.osd.7.smithi190.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fc0d1556a00] 2023-11-01T04:16:57.824 INFO:tasks.ceph.osd.7.smithi190.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-01T04:16:57.824 INFO:tasks.ceph.osd.7.smithi190.stderr: 2023-11-01T04:16:57.831 INFO:teuthology.orchestra.run.smithi153.stderr:nodeep-scrub is set 2023-11-01T04:16:57.839 INFO:tasks.rados.rados.0.smithi190.stdout:267: done (5 left) 2023-11-01T04:16:57.839 INFO:tasks.rados.rados.0.smithi190.stdout:268: read oid 38 snap -1 2023-11-01T04:16:57.839 INFO:tasks.rados.rados.0.smithi190.stdout:268: expect (ObjNum 37 snap 0 seq_num 37) 2023-11-01T04:16:57.840 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 10 v 54 (ObjNum 66 snap 6 seq_num 66) dirty exists 2023-11-01T04:16:57.840 INFO:tasks.rados.rados.0.smithi190.stdout:262: finishing rollback tid 1 to smithi19039307-40 2023-11-01T04:16:57.840 INFO:tasks.rados.rados.0.smithi190.stdout:269: read oid 4 snap -1 2023-11-01T04:16:57.840 INFO:tasks.rados.rados.0.smithi190.stdout:269: expect (ObjNum 3 snap 0 seq_num 3) 2023-11-01T04:16:57.840 INFO:tasks.rados.rados.0.smithi190.stdout:265: done (6 left) 2023-11-01T04:16:57.840 INFO:tasks.rados.rados.0.smithi190.stdout:270: read oid 19 snap -1 2023-11-01T04:16:57.841 INFO:tasks.rados.rados.0.smithi190.stdout:270: expect (ObjNum 89 snap 20 seq_num 89) 2023-11-01T04:16:57.841 INFO:tasks.rados.rados.0.smithi190.stdout:271: snap_remove snap 16 2023-11-01T04:16:57.907 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:57.908+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:57.945 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:16:57.948 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:16:57.948 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['config', 'set', 'filestore_inject_stall', '3'] 2023-11-01T04:16:57.950 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: exception: [Errno 104] Connection reset by peer 2023-11-01T04:16:57.954 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:16:57.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-01T04:16:57.996 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:57.996+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:58.071 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:58.065+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:58.107 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:16:58.110 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:16:58.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-01T04:16:58.173 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:16:58.172+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:16:58.177 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-01T04:16:58.259 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:16:58.262 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:16:58.263 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-01T04:16:58.274 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:16:58.273+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:16:58.334 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:16:58.334 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-11-01T04:16:58.355 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:16:58.355 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-01T04:16:58.359 INFO:tasks.ceph.osd.7.smithi190.stderr:daemon-helper: command crashed with signal 6 2023-11-01T04:16:58.374 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:58.373+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:58.375 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:16:58.375 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-01T04:16:58.389 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:58.389 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:58.394 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:16:58.394 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-01T04:16:58.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-01T04:16:58.411 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:16:58.411 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-01T04:16:58.431 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:16:58.432 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-01T04:16:58.452 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:16:58.453 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-01T04:16:58.492 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:58.489+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:58.566 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:58.566 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:58.576 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:16:58.576+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:16:58.581 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-01T04:16:58.677 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:16:58.675+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:16:58.743 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:58.744 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:58.758 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-01T04:16:58.778 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:16:58.777+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:16:58.842 INFO:tasks.rados.rados.0.smithi190.stdout:271: done (7 left) 2023-11-01T04:16:58.842 INFO:tasks.rados.rados.0.smithi190.stdout:272: copy_from oid 6 from oid 48 current snap is 24 2023-11-01T04:16:58.843 INFO:tasks.rados.rados.0.smithi190.stdout:262: finishing rollback tid 2 to smithi19039307-40 2023-11-01T04:16:58.843 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 40 v 70 (ObjNum 39 snap 0 seq_num 39) dirty exists 2023-11-01T04:16:58.843 INFO:tasks.rados.rados.0.smithi190.stdout:263: expect (ObjNum 53 snap 1 seq_num 53) 2023-11-01T04:16:58.879 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:58.873+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:58.886 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:58.887 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:58.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-01T04:16:58.913 INFO:tasks.rados.rados.0.smithi190.stdout:264: finishing write tid 1 to smithi19039307-46 2023-11-01T04:16:58.913 INFO:tasks.rados.rados.0.smithi190.stdout:264: finishing write tid 2 to smithi19039307-46 2023-11-01T04:16:58.914 INFO:tasks.rados.rados.0.smithi190.stdout:262: done (7 left) 2023-11-01T04:16:58.914 INFO:tasks.rados.rados.0.smithi190.stdout:263: done (6 left) 2023-11-01T04:16:58.914 INFO:tasks.rados.rados.0.smithi190.stdout:273: read oid 31 snap -1 2023-11-01T04:16:58.914 INFO:tasks.rados.rados.0.smithi190.stdout:273: expect (ObjNum 86 snap 19 seq_num 86) 2023-11-01T04:16:58.914 INFO:tasks.rados.rados.0.smithi190.stdout:274: snap_remove snap 12 2023-11-01T04:16:58.980 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:58.977+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:59.035 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:59.035 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:59.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-01T04:16:59.080 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:59.080+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:59.171 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:16:59.172 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:16:59.181 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:59.177+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:59.185 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-01T04:16:59.280 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:16:59.316 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:16:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:659 3.cs0 3.511263cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-11-01T04:16:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:58.843911+0000", 2023-11-01T04:16:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.47286662899999998, 2023-11-01T04:16:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.47290901000000002, 2023-11-01T04:16:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "waiting for peered", 2023-11-01T04:16:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 659 2023-11-01T04:16:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.843911+0000", 2023-11-01T04:16:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.843911+0000", 2023-11-01T04:16:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.843914+0000", 2023-11-01T04:16:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2760000000000001e-06 2023-11-01T04:16:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.843948+0000", 2023-11-01T04:16:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.3089000000000001e-05 2023-11-01T04:16:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.843982+0000", 2023-11-01T04:16:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.3961999999999998e-05 2023-11-01T04:16:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.844009+0000", 2023-11-01T04:16:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7498999999999999e-05 2023-11-01T04:16:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.247668+0000", 2023-11-01T04:16:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.40365936499999999 2023-11-01T04:16:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2023-11-01T04:16:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.247675+0000", 2023-11-01T04:16:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6699999999999997e-06 2023-11-01T04:16:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:660 3.cs0 3.511263cc (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2023-11-01T04:16:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:58.844062+0000", 2023-11-01T04:16:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.47271618500000001, 2023-11-01T04:16:59.332 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.472893061, 2023-11-01T04:16:59.332 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.332 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:59.332 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 660 2023-11-01T04:16:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.844062+0000", 2023-11-01T04:16:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.844062+0000", 2023-11-01T04:16:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.337 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.337 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.844063+0000", 2023-11-01T04:16:59.337 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6029999999999999e-06 2023-11-01T04:16:59.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.844070+0000", 2023-11-01T04:16:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1960000000000002e-06 2023-11-01T04:16:59.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.339 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.844094+0000", 2023-11-01T04:16:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3159999999999998e-05 2023-11-01T04:16:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.844115+0000", 2023-11-01T04:16:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1685e-05 2023-11-01T04:16:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-11-01T04:16:59.343 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 2 2023-11-01T04:16:59.343 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:59.344 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-01T04:16:59.383 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:16:59.381+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:16:59.468 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:59.468 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:16:59.468 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2023-11-01T04:16:59.468 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2023-11-01T04:16:59.468 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:59.482 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-01T04:16:59.483 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:16:59.484+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:16:59.584 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:16:59.583+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:16:59.602 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:59.602 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2023-11-01T04:16:59.602 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2023-11-01T04:16:59.603 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:16:59.603 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.603 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:188 3.fs0 3:f81e983c:::smithi19039307-34:head [create,setxattr XNeZS[lOPtVca\\Pa[\\rdfbinb[bcjhtYSpmcafZm (1928) in=40b,setxattr TfflYYqPnps]lo^qSlp___TitNdd`]kaPTVOTcsh (1648) in=40b,setxattr TNb_onpY[j_dkhfmnWtm]Yktg]]RNtbQe[OUTNlX (770) in=40b,setxattr ZcgnkhY\\orTYaZh]PPldj`SNq`p\\cX`k]h\\bZY_Y (700) in=40b,setxattr gPQVnqhpsVhW_kq`QnafXrlRZtZ\\YnRftgQTishR (595) in=40b,setxattr U`XqYi[WhoqnpSPRc\\RhnppZp^qY\\qUfjpqs[r^a (1892) in=40b,setxattr cTmhd`pWgrRf_VfVgOtgeOP^ShP]lsOtpoTj[\\gl (418) in=40b,setxattr aa_taYTfslg`lRRrOkkraN\\XiW^V\\ljl]NqXin`a (314) in=40b,setxattr ibl]`igtgVThY^Qse\\okYYUtTmolX\\W]p]Q[P\\il (1403) in=40b,setxattr \\ZabSO[UUqeZa`hddNh[ZjaRRgUZoQtkn\\QdsX]W (1541) in=40b,setxattr jUkQUPTNbo`rZNmYlUmhb]ojc]aeheoVRfp]bPWf (5052) in=40b,setxattr Ofj`oQ^cR]da^SPsTaXmmPse\\OZ`VSqQ]U`efsZl (419) in=40b,setxattr a[cdkWos\\OrUVPciUaa^_^`lWVTScffiWkd\\[kgf (1261) in=40b,setxattr jXb]\\WhbatdZ[qTTiZO]kPd^`l[kajOYiQ[iqSYc (244) in=40b,setxattr \\im]jQhbanUn\\_^h`jaU]W`ReaZN_je[]kgg (1578) in=36b,setxattr ZbosoRORS^_de]rN^_daOiha\\Q\\eRae^[sQUjXsV (9120) in=40b,setxattr gcn]XpZUkOZr\\pkrVVZpYiR\\alh_PaZa_S[PYcn] (384) in=40b,setxattr \\^qUOaNXSTjtmafhfbo\\gseTTkV`mgnacYq^bkcc (34) in=40b,setxattr iUa^mOQ]TqjVSUU_nXfY]f`NjljiiaYbrm[o]aes (1611) in=40b,setxattr tXaQPQbc^aiV[^reVOPpib^caoWf^XgqilO^_g[k (1016) in=40b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:59.603 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.964946+0000", 2023-11-01T04:16:59.603 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 42.635150209000003, 2023-11-01T04:16:59.604 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.309617849, 2023-11-01T04:16:59.604 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.604 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:16:59.604 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:59.605 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:59.605 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:59.605 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 188 2023-11-01T04:16:59.605 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.605 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.606 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.606 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.606 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.964946+0000", 2023-11-01T04:16:59.606 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.607 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.607 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.607 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.964946+0000", 2023-11-01T04:16:59.607 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.608 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.608 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.608 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.964948+0000", 2023-11-01T04:16:59.608 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.824e-06 2023-11-01T04:16:59.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.608 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.609 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.609 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.965026+0000", 2023-11-01T04:16:59.609 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.8151000000000004e-05 2023-11-01T04:16:59.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.609 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.610 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.610 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.965047+0000", 2023-11-01T04:16:59.610 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0891e-05 2023-11-01T04:16:59.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.610 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.611 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.611 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.965062+0000", 2023-11-01T04:16:59.611 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5310000000000001e-05 2023-11-01T04:16:59.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.611 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.612 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.612 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.965154+0000", 2023-11-01T04:16:59.612 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2163999999999994e-05 2023-11-01T04:16:59.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.613 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.613 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2023-11-01T04:16:59.613 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.965165+0000", 2023-11-01T04:16:59.613 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0703000000000001e-05 2023-11-01T04:16:59.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.613 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.614 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.614 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.167756+0000", 2023-11-01T04:16:59.614 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20259100399999999 2023-11-01T04:16:59.614 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.614 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.615 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:59.615 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.167952+0000", 2023-11-01T04:16:59.615 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00019548100000000001 2023-11-01T04:16:59.615 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.615 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.616 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:59.616 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.169709+0000", 2023-11-01T04:16:59.616 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0017572989999999999 2023-11-01T04:16:59.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.616 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.617 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:59.617 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.171668+0000", 2023-11-01T04:16:59.617 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0019592659999999999 2023-11-01T04:16:59.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.617 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.617 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:16:59.618 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.274435+0000", 2023-11-01T04:16:59.618 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10276656000000001 2023-11-01T04:16:59.618 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.618 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.618 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.619 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.274564+0000", 2023-11-01T04:16:59.619 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012919599999999999 2023-11-01T04:16:59.619 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.619 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.620 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.620 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.620 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 37/31 ECSubWriteReply(tid=9, last_complete=37'4, committed=1, applied=1))", 2023-11-01T04:16:59.620 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:17.277556+0000", 2023-11-01T04:16:59.621 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 42.322539843999998, 2023-11-01T04:16:59.621 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20033701600000001, 2023-11-01T04:16:59.621 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.621 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:59.621 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.622 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.622 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.622 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.277556+0000", 2023-11-01T04:16:59.622 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.622 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.623 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.623 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.623 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.277556+0000", 2023-11-01T04:16:59.623 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.624 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.624 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.624 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.277558+0000", 2023-11-01T04:16:59.624 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.217e-06 2023-11-01T04:16:59.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.625 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.625 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.625 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.277571+0000", 2023-11-01T04:16:59.625 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3032e-05 2023-11-01T04:16:59.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.625 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.626 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.626 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.277593+0000", 2023-11-01T04:16:59.626 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2160000000000001e-05 2023-11-01T04:16:59.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.626 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.627 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.627 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.377685+0000", 2023-11-01T04:16:59.627 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10009243700000001 2023-11-01T04:16:59.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.627 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.627 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.628 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.477859+0000", 2023-11-01T04:16:59.628 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10017401500000001 2023-11-01T04:16:59.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.628 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.628 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.629 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.477894+0000", 2023-11-01T04:16:59.629 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4155000000000001e-05 2023-11-01T04:16:59.629 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.629 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.629 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.630 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.630 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(osd.11.22:2 3.cs0 3:33c6488a:::smithi19039307-6:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:16:59.630 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:17.378858+0000", 2023-11-01T04:16:59.630 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 42.221238452999998, 2023-11-01T04:16:59.630 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.203072057, 2023-11-01T04:16:59.631 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.631 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:59.631 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.631 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.631 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.632 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378858+0000", 2023-11-01T04:16:59.632 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.632 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.632 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.633 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378858+0000", 2023-11-01T04:16:59.633 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.633 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.633 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.634 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378860+0000", 2023-11-01T04:16:59.634 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5339999999999998e-06 2023-11-01T04:16:59.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.634 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.634 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.635 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378872+0000", 2023-11-01T04:16:59.635 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1365999999999999e-05 2023-11-01T04:16:59.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.635 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.636 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378896+0000", 2023-11-01T04:16:59.636 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3935999999999999e-05 2023-11-01T04:16:59.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.636 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.636 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378909+0000", 2023-11-01T04:16:59.637 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3523000000000001e-05 2023-11-01T04:16:59.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.637 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.637 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.379027+0000", 2023-11-01T04:16:59.638 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000117963 2023-11-01T04:16:59.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.638 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:59.638 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.379186+0000", 2023-11-01T04:16:59.638 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000158939 2023-11-01T04:16:59.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.639 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.639 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.581930+0000", 2023-11-01T04:16:59.639 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.202743796 2023-11-01T04:16:59.640 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.640 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.640 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.640 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.640 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 41/31 ECSubWriteReply(tid=12, last_complete=41'6, committed=1, applied=1))", 2023-11-01T04:16:59.641 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.240162+0000", 2023-11-01T04:16:59.641 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 38.359934422000002, 2023-11-01T04:16:59.641 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19880108899999999, 2023-11-01T04:16:59.641 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.641 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:59.642 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.642 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.642 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.642 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.240162+0000", 2023-11-01T04:16:59.642 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.643 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.643 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.643 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.240162+0000", 2023-11-01T04:16:59.643 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.644 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.644 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.240163+0000", 2023-11-01T04:16:59.644 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4200000000000004e-07 2023-11-01T04:16:59.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.645 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.645 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.645 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.240168+0000", 2023-11-01T04:16:59.645 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4729999999999999e-06 2023-11-01T04:16:59.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.645 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.646 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.646 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.240181+0000", 2023-11-01T04:16:59.646 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2639999999999999e-05 2023-11-01T04:16:59.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.646 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.647 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.647 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.340265+0000", 2023-11-01T04:16:59.647 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100084464 2023-11-01T04:16:59.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.647 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.647 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.438933+0000", 2023-11-01T04:16:59.648 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.098667650999999995 2023-11-01T04:16:59.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.648 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.649 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.438963+0000", 2023-11-01T04:16:59.649 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9918999999999999e-05 2023-11-01T04:16:59.649 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.649 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.649 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.650 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.650 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(osd.14.22:2 3.cs0 3:33c6488a:::smithi19039307-6:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:16:59.650 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.134094+0000", 2023-11-01T04:16:59.650 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 33.466002109000001, 2023-11-01T04:16:59.650 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.35681233299999998, 2023-11-01T04:16:59.651 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.651 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:59.651 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.651 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.651 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.651 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134094+0000", 2023-11-01T04:16:59.652 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.652 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.652 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.652 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134094+0000", 2023-11-01T04:16:59.652 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.653 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.653 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.653 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134096+0000", 2023-11-01T04:16:59.653 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1919999999999999e-06 2023-11-01T04:16:59.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.654 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.654 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.654 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134108+0000", 2023-11-01T04:16:59.654 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1508000000000001e-05 2023-11-01T04:16:59.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.655 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.655 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.655 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134129+0000", 2023-11-01T04:16:59.655 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0968e-05 2023-11-01T04:16:59.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.655 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.656 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.656 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134151+0000", 2023-11-01T04:16:59.656 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2480000000000002e-05 2023-11-01T04:16:59.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.656 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.656 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.657 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134288+0000", 2023-11-01T04:16:59.657 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013673799999999999 2023-11-01T04:16:59.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.657 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.657 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:59.657 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134380+0000", 2023-11-01T04:16:59.658 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.1555e-05 2023-11-01T04:16:59.658 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.658 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.658 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.658 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.490907+0000", 2023-11-01T04:16:59.659 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.35652689199999998 2023-11-01T04:16:59.659 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.659 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.659 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.659 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.659 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.660 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.cs0 45/31 ECSubReadReply(tid=18, attrs_read=0))", 2023-11-01T04:16:59.660 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.135072+0000", 2023-11-01T04:16:59.660 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 33.465024794999998, 2023-11-01T04:16:59.660 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20028368799999999, 2023-11-01T04:16:59.660 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.660 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:59.661 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.661 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.661 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.661 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135072+0000", 2023-11-01T04:16:59.661 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.662 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.662 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.662 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135072+0000", 2023-11-01T04:16:59.662 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.663 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.663 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135072+0000", 2023-11-01T04:16:59.663 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5199999999999995e-07 2023-11-01T04:16:59.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.664 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.664 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135109+0000", 2023-11-01T04:16:59.664 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6155999999999997e-05 2023-11-01T04:16:59.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.664 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.665 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.665 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135118+0000", 2023-11-01T04:16:59.665 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7399999999999999e-06 2023-11-01T04:16:59.665 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.665 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.665 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.666 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135123+0000", 2023-11-01T04:16:59.666 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.8160000000000001e-06 2023-11-01T04:16:59.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.666 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.666 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.667 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.335285+0000", 2023-11-01T04:16:59.667 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20016213599999999 2023-11-01T04:16:59.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.667 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.667 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.668 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.335355+0000", 2023-11-01T04:16:59.668 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9888000000000006e-05 2023-11-01T04:16:59.668 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.668 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.668 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.669 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.669 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.cs0 46/31 ECSubReadReply(tid=18, attrs_read=0))", 2023-11-01T04:16:59.669 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.135473+0000", 2023-11-01T04:16:59.669 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 33.464622962999997, 2023-11-01T04:16:59.669 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19999888900000001, 2023-11-01T04:16:59.670 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.670 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:59.670 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.670 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.670 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.671 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135473+0000", 2023-11-01T04:16:59.671 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.671 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.671 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.671 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.672 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135473+0000", 2023-11-01T04:16:59.672 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.672 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.672 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135474+0000", 2023-11-01T04:16:59.673 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1200000000000002e-07 2023-11-01T04:16:59.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.673 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.673 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135586+0000", 2023-11-01T04:16:59.673 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000112274 2023-11-01T04:16:59.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.674 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.674 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135602+0000", 2023-11-01T04:16:59.674 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5044e-05 2023-11-01T04:16:59.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.675 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.675 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135611+0000", 2023-11-01T04:16:59.675 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0210000000000005e-06 2023-11-01T04:16:59.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.678 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.678 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.679 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.335430+0000", 2023-11-01T04:16:59.679 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19981938099999999 2023-11-01T04:16:59.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.679 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.680 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.335472+0000", 2023-11-01T04:16:59.680 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2357000000000001e-05 2023-11-01T04:16:59.680 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.680 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.680 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.681 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.681 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s2 46/31 ECSubWrite(tid=235, reqid=client.4559.0:329, at_version=46'35, trim_to=0'0, roll_forward_to=45'33))", 2023-11-01T04:16:59.681 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.179884+0000", 2023-11-01T04:16:59.681 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 33.420211895999998, 2023-11-01T04:16:59.682 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20567904100000001, 2023-11-01T04:16:59.682 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.682 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:59.682 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.682 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.682 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.683 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.179884+0000", 2023-11-01T04:16:59.683 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.683 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.683 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.684 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.179884+0000", 2023-11-01T04:16:59.684 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.684 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.684 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.684 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.684 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.179885+0000", 2023-11-01T04:16:59.685 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.74e-07 2023-11-01T04:16:59.685 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.685 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.685 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.685 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.180252+0000", 2023-11-01T04:16:59.685 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00036695899999999998 2023-11-01T04:16:59.686 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.686 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.686 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.686 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.180264+0000", 2023-11-01T04:16:59.686 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2850999999999999e-05 2023-11-01T04:16:59.686 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.687 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.687 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.687 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.280394+0000", 2023-11-01T04:16:59.687 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100129722 2023-11-01T04:16:59.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.688 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.688 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.688 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.280540+0000", 2023-11-01T04:16:59.688 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014556000000000001 2023-11-01T04:16:59.688 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.688 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.689 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:59.689 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.280557+0000", 2023-11-01T04:16:59.689 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7543000000000001e-05 2023-11-01T04:16:59.689 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.689 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.689 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:59.690 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.385509+0000", 2023-11-01T04:16:59.690 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104951378 2023-11-01T04:16:59.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.690 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.690 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.691 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.385563+0000", 2023-11-01T04:16:59.691 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4854e-05 2023-11-01T04:16:59.691 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.691 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.691 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.692 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.692 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.692 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.8s5 46/31 ECSubWrite(tid=30, reqid=client.4559.0:341, at_version=46'22, trim_to=0'0, roll_forward_to=43'14))", 2023-11-01T04:16:59.692 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.266862+0000", 2023-11-01T04:16:59.692 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 33.333233874000001, 2023-11-01T04:16:59.692 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20203428600000001, 2023-11-01T04:16:59.693 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.693 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:59.693 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.693 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.693 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.693 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266862+0000", 2023-11-01T04:16:59.694 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.694 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.694 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.694 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.694 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266862+0000", 2023-11-01T04:16:59.695 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.695 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.695 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.695 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.695 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266865+0000", 2023-11-01T04:16:59.695 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2840000000000001e-06 2023-11-01T04:16:59.696 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.696 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.696 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.696 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266897+0000", 2023-11-01T04:16:59.696 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2743000000000001e-05 2023-11-01T04:16:59.697 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.697 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.697 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.697 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266920+0000", 2023-11-01T04:16:59.697 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2878999999999999e-05 2023-11-01T04:16:59.698 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.698 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.698 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.698 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266939+0000", 2023-11-01T04:16:59.698 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8185999999999998e-05 2023-11-01T04:16:59.698 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.699 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.699 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.699 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.467334+0000", 2023-11-01T04:16:59.699 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20039552699999999 2023-11-01T04:16:59.699 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.700 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.700 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:59.700 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.467350+0000", 2023-11-01T04:16:59.700 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5682999999999999e-05 2023-11-01T04:16:59.700 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.700 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.468855+0000", 2023-11-01T04:16:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001504825 2023-11-01T04:16:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.701 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.468897+0000", 2023-11-01T04:16:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2159000000000001e-05 2023-11-01T04:16:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.702 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:347 3.fs0 3:f81e983c:::smithi19039307-34:c [read 0~557056,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.032054+0000", 2023-11-01T04:16:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 32.568042173000002, 2023-11-01T04:16:59.703 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20755394999999999, 2023-11-01T04:16:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:59.704 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 347 2023-11-01T04:16:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032054+0000", 2023-11-01T04:16:59.705 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032054+0000", 2023-11-01T04:16:59.706 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032058+0000", 2023-11-01T04:16:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4740000000000001e-06 2023-11-01T04:16:59.707 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032074+0000", 2023-11-01T04:16:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.679e-05 2023-11-01T04:16:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.708 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032109+0000", 2023-11-01T04:16:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4309e-05 2023-11-01T04:16:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.709 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032128+0000", 2023-11-01T04:16:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8856999999999999e-05 2023-11-01T04:16:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.710 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032256+0000", 2023-11-01T04:16:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000128866 2023-11-01T04:16:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.711 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2023-11-01T04:16:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032268+0000", 2023-11-01T04:16:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1649e-05 2023-11-01T04:16:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.712 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.132806+0000", 2023-11-01T04:16:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10053780399999999 2023-11-01T04:16:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.132905+0000", 2023-11-01T04:16:59.713 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.9326000000000004e-05 2023-11-01T04:16:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.239608+0000", 2023-11-01T04:16:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.106702875 2023-11-01T04:16:59.714 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.715 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:348 3.fs0 3:f81e983c:::smithi19039307-34:c [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.032168+0000", 2023-11-01T04:16:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 32.567927877999999, 2023-11-01T04:16:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30608133399999998, 2023-11-01T04:16:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:59.716 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 348 2023-11-01T04:16:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.717 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032168+0000", 2023-11-01T04:16:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.718 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032168+0000", 2023-11-01T04:16:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.719 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032169+0000", 2023-11-01T04:16:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8199999999999999e-07 2023-11-01T04:16:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.720 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.721 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032174+0000", 2023-11-01T04:16:59.721 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1379999999999998e-06 2023-11-01T04:16:59.721 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.721 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.721 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.721 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032186+0000", 2023-11-01T04:16:59.722 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1229e-05 2023-11-01T04:16:59.722 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.722 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.722 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.722 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032193+0000", 2023-11-01T04:16:59.723 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2749999999999998e-06 2023-11-01T04:16:59.723 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.723 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.723 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.723 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032361+0000", 2023-11-01T04:16:59.724 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016853199999999999 2023-11-01T04:16:59.724 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.724 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.724 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:59.724 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032368+0000", 2023-11-01T04:16:59.724 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.686e-06 2023-11-01T04:16:59.725 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.725 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.725 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.725 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.135190+0000", 2023-11-01T04:16:59.725 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10282246 2023-11-01T04:16:59.726 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.726 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.726 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:59.726 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.135261+0000", 2023-11-01T04:16:59.726 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0296999999999995e-05 2023-11-01T04:16:59.727 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.727 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.727 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.727 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.338250+0000", 2023-11-01T04:16:59.727 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20298903500000001 2023-11-01T04:16:59.727 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.728 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.728 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.728 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.728 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.728 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:349 3.fs0 3:f81e983c:::smithi19039307-34:c [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:16:59.729 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.032204+0000", 2023-11-01T04:16:59.729 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 32.567892419000003, 2023-11-01T04:16:59.729 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.61100175800000001, 2023-11-01T04:16:59.729 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.729 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:59.729 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:16:59.730 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:16:59.730 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:16:59.730 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 349 2023-11-01T04:16:59.730 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.730 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.731 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.731 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.731 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032204+0000", 2023-11-01T04:16:59.731 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.731 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.732 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.732 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.732 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032204+0000", 2023-11-01T04:16:59.732 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.732 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.733 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.733 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.733 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032204+0000", 2023-11-01T04:16:59.733 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2200000000000004e-07 2023-11-01T04:16:59.733 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.734 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.734 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.734 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032209+0000", 2023-11-01T04:16:59.734 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5979999999999999e-06 2023-11-01T04:16:59.734 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.735 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.735 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.735 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032217+0000", 2023-11-01T04:16:59.735 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9019999999999999e-06 2023-11-01T04:16:59.735 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.736 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.736 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.736 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032223+0000", 2023-11-01T04:16:59.736 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7180000000000001e-06 2023-11-01T04:16:59.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.736 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.737 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.737 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.132534+0000", 2023-11-01T04:16:59.737 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10031093200000001 2023-11-01T04:16:59.737 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.737 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.737 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:16:59.738 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.132550+0000", 2023-11-01T04:16:59.738 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6269999999999998e-05 2023-11-01T04:16:59.738 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.738 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.738 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.738 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.339795+0000", 2023-11-01T04:16:59.739 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20724526200000001 2023-11-01T04:16:59.739 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.739 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.739 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:59.739 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.340097+0000", 2023-11-01T04:16:59.740 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00030174899999999997 2023-11-01T04:16:59.740 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.740 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.740 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.740 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.643206+0000", 2023-11-01T04:16:59.740 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30310880499999998 2023-11-01T04:16:59.741 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.741 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.741 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.741 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.741 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.742 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.fs0 47/31 ECSubReadReply(tid=21, attrs_read=0))", 2023-11-01T04:16:59.742 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.349538+0000", 2023-11-01T04:16:59.742 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 32.250558423999998, 2023-11-01T04:16:59.742 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.29371594899999998, 2023-11-01T04:16:59.742 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.742 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:59.743 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.743 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.743 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.743 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349538+0000", 2023-11-01T04:16:59.743 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.743 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.744 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.744 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.744 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349538+0000", 2023-11-01T04:16:59.744 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.744 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.745 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.745 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.745 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349539+0000", 2023-11-01T04:16:59.745 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5370000000000001e-06 2023-11-01T04:16:59.745 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.746 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.746 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.746 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349659+0000", 2023-11-01T04:16:59.746 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011938 2023-11-01T04:16:59.746 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.746 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.747 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.747 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349686+0000", 2023-11-01T04:16:59.747 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.728e-05 2023-11-01T04:16:59.747 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.747 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.748 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.748 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349702+0000", 2023-11-01T04:16:59.748 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5693e-05 2023-11-01T04:16:59.748 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.748 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.749 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.749 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.642479+0000", 2023-11-01T04:16:59.749 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.29277763699999998 2023-11-01T04:16:59.749 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.749 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.750 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.750 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.643254+0000", 2023-11-01T04:16:59.750 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00077442199999999998 2023-11-01T04:16:59.750 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.750 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.750 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.751 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.751 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.751 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(osd.13.22:6 3.fs0 3:f81e983c:::smithi19039307-34:head [assert-version v12,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:16:59.751 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:48.189226+0000", 2023-11-01T04:16:59.751 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 11.410870284, 2023-11-01T04:16:59.752 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30167290200000002, 2023-11-01T04:16:59.752 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.752 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:16:59.752 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.752 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.753 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.753 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.189226+0000", 2023-11-01T04:16:59.753 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.753 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.753 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.753 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.754 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.189226+0000", 2023-11-01T04:16:59.754 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.754 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.754 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.754 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.755 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.189229+0000", 2023-11-01T04:16:59.755 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4729999999999999e-06 2023-11-01T04:16:59.755 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.755 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.755 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.756 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.189239+0000", 2023-11-01T04:16:59.756 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0036e-05 2023-11-01T04:16:59.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.756 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.756 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.756 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.189260+0000", 2023-11-01T04:16:59.757 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1206999999999999e-05 2023-11-01T04:16:59.757 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.757 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.757 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.757 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.189277+0000", 2023-11-01T04:16:59.757 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6946e-05 2023-11-01T04:16:59.758 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.758 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.758 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.758 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.197102+0000", 2023-11-01T04:16:59.758 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0078254740000000007 2023-11-01T04:16:59.759 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.759 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.759 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:16:59.759 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.197404+0000", 2023-11-01T04:16:59.759 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00030222199999999999 2023-11-01T04:16:59.759 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.760 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.760 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.760 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.490899+0000", 2023-11-01T04:16:59.760 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.29349454400000002 2023-11-01T04:16:59.760 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.761 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.761 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.761 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.761 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.761 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.fs0 68/31 ECSubReadReply(tid=33, attrs_read=0))", 2023-11-01T04:16:59.761 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:48.198904+0000", 2023-11-01T04:16:59.762 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 11.40119213, 2023-11-01T04:16:59.762 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.191258976, 2023-11-01T04:16:59.762 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.762 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:59.762 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.762 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.763 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.763 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.198904+0000", 2023-11-01T04:16:59.763 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.763 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.763 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.764 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.764 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.198904+0000", 2023-11-01T04:16:59.764 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.764 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.764 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.764 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.765 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.198909+0000", 2023-11-01T04:16:59.765 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1590000000000001e-06 2023-11-01T04:16:59.765 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.765 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.765 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.765 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.199297+0000", 2023-11-01T04:16:59.766 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00038786700000000002 2023-11-01T04:16:59.766 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.766 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.766 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.766 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.199325+0000", 2023-11-01T04:16:59.766 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7457000000000002e-05 2023-11-01T04:16:59.767 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.767 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.767 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.767 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.199338+0000", 2023-11-01T04:16:59.767 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3623e-05 2023-11-01T04:16:59.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.768 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.768 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.768 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.390093+0000", 2023-11-01T04:16:59.768 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.190754902 2023-11-01T04:16:59.768 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.769 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.769 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.769 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.390163+0000", 2023-11-01T04:16:59.769 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9968000000000005e-05 2023-11-01T04:16:59.769 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.770 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.770 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.770 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.770 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.770 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.fs0 68/31 ECSubReadReply(tid=33, attrs_read=0))", 2023-11-01T04:16:59.770 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:48.299049+0000", 2023-11-01T04:16:59.771 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 11.301047147, 2023-11-01T04:16:59.771 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.191882055, 2023-11-01T04:16:59.771 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.771 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:59.771 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.772 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.772 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.772 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.299049+0000", 2023-11-01T04:16:59.772 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.772 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.773 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.773 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.773 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.299049+0000", 2023-11-01T04:16:59.773 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.773 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.774 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.774 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.299051+0000", 2023-11-01T04:16:59.774 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1430000000000001e-06 2023-11-01T04:16:59.774 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.774 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.774 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.775 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.299433+0000", 2023-11-01T04:16:59.775 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000382154 2023-11-01T04:16:59.775 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.775 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.775 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.776 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.299472+0000", 2023-11-01T04:16:59.776 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8136000000000002e-05 2023-11-01T04:16:59.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.776 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.776 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.776 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.299493+0000", 2023-11-01T04:16:59.777 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1526999999999999e-05 2023-11-01T04:16:59.777 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.777 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.777 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.777 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.490427+0000", 2023-11-01T04:16:59.777 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19093384299999999 2023-11-01T04:16:59.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.778 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.778 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.778 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.490931+0000", 2023-11-01T04:16:59.778 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00050425199999999996 2023-11-01T04:16:59.778 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.779 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.779 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.779 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.779 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s2 69/31 ECSubWrite(tid=324, reqid=client.4559.0:569, at_version=69'78, trim_to=0'0, roll_forward_to=68'74))", 2023-11-01T04:16:59.779 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:49.613254+0000", 2023-11-01T04:16:59.780 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 9.9868423699999997, 2023-11-01T04:16:59.780 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20193297099999999, 2023-11-01T04:16:59.780 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.780 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:59.780 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.781 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.781 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.781 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613254+0000", 2023-11-01T04:16:59.781 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.781 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.782 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.782 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.782 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613254+0000", 2023-11-01T04:16:59.782 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.782 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.783 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.783 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613255+0000", 2023-11-01T04:16:59.783 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6700000000000003e-07 2023-11-01T04:16:59.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.783 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.784 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.784 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613466+0000", 2023-11-01T04:16:59.784 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00021175600000000001 2023-11-01T04:16:59.784 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.784 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.785 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.785 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613485+0000", 2023-11-01T04:16:59.785 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8499999999999999e-05 2023-11-01T04:16:59.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.785 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.785 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.786 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613495+0000", 2023-11-01T04:16:59.786 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7200000000000001e-06 2023-11-01T04:16:59.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.786 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.786 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.787 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.714261+0000", 2023-11-01T04:16:59.787 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10076626399999999 2023-11-01T04:16:59.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.787 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.787 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:59.787 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.714278+0000", 2023-11-01T04:16:59.788 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7462999999999999e-05 2023-11-01T04:16:59.788 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.788 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.788 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:59.788 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.815118+0000", 2023-11-01T04:16:59.789 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100840022 2023-11-01T04:16:59.789 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.789 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.789 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.789 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.815187+0000", 2023-11-01T04:16:59.790 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8678999999999998e-05 2023-11-01T04:16:59.790 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.790 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.790 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.791 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.791 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s2 69/31 ECSubWrite(tid=325, reqid=client.4559.0:570, at_version=69'79, trim_to=0'0, roll_forward_to=68'74))", 2023-11-01T04:16:59.791 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:49.613511+0000", 2023-11-01T04:16:59.791 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 9.9865857469999995, 2023-11-01T04:16:59.791 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20304860599999999, 2023-11-01T04:16:59.791 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.792 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:59.792 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.792 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.792 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613511+0000", 2023-11-01T04:16:59.793 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.793 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.793 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.793 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613511+0000", 2023-11-01T04:16:59.793 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.794 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.794 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.794 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.794 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613511+0000", 2023-11-01T04:16:59.794 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0999999999999999e-07 2023-11-01T04:16:59.794 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.795 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.795 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.795 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613647+0000", 2023-11-01T04:16:59.795 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000136124 2023-11-01T04:16:59.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.795 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.796 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.796 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613662+0000", 2023-11-01T04:16:59.796 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5146e-05 2023-11-01T04:16:59.796 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.796 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.797 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.797 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.713796+0000", 2023-11-01T04:16:59.797 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100133378 2023-11-01T04:16:59.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.797 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.797 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.798 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.814080+0000", 2023-11-01T04:16:59.798 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10028453499999999 2023-11-01T04:16:59.798 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.798 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.798 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:59.798 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.814096+0000", 2023-11-01T04:16:59.799 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6030000000000001e-05 2023-11-01T04:16:59.799 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.799 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.799 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:59.799 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.816487+0000", 2023-11-01T04:16:59.799 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002391121 2023-11-01T04:16:59.800 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.800 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.800 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.800 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.816559+0000", 2023-11-01T04:16:59.800 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1761999999999995e-05 2023-11-01T04:16:59.801 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.801 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.801 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.801 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.801 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.802 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.6s5 72/68 ECSubWrite(tid=102, reqid=client.4559.0:608, at_version=72'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-01T04:16:59.802 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:52.898759+0000", 2023-11-01T04:16:59.802 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.7013370810000001, 2023-11-01T04:16:59.802 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30376394600000001, 2023-11-01T04:16:59.802 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.802 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:59.803 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.803 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.803 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.803 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.898759+0000", 2023-11-01T04:16:59.803 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.803 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.804 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.804 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.804 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.898759+0000", 2023-11-01T04:16:59.804 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.805 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.805 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.805 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.898760+0000", 2023-11-01T04:16:59.805 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0800000000000004e-07 2023-11-01T04:16:59.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.805 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.806 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.806 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.898794+0000", 2023-11-01T04:16:59.806 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.3899000000000002e-05 2023-11-01T04:16:59.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.806 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.807 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.807 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.898805+0000", 2023-11-01T04:16:59.807 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1467e-05 2023-11-01T04:16:59.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.807 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.808 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.808 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.898814+0000", 2023-11-01T04:16:59.808 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.2570000000000007e-06 2023-11-01T04:16:59.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.808 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.809 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.809 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.201072+0000", 2023-11-01T04:16:59.809 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30225848500000002 2023-11-01T04:16:59.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.809 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.809 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:16:59.810 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.201088+0000", 2023-11-01T04:16:59.810 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5549999999999999e-05 2023-11-01T04:16:59.810 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.810 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.810 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:16:59.811 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.202443+0000", 2023-11-01T04:16:59.811 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001355057 2023-11-01T04:16:59.811 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.811 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.811 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.811 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.202523+0000", 2023-11-01T04:16:59.812 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0422999999999999e-05 2023-11-01T04:16:59.812 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.812 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.812 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.812 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.813 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 72/31 ECSubWriteReply(tid=34, last_complete=72'20, committed=1, applied=1))", 2023-11-01T04:16:59.813 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:52.905947+0000", 2023-11-01T04:16:59.813 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.6941494810000002, 2023-11-01T04:16:59.813 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20027774300000001, 2023-11-01T04:16:59.813 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.814 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:16:59.814 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.814 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.814 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.814 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.905947+0000", 2023-11-01T04:16:59.814 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.815 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.815 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.815 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.905947+0000", 2023-11-01T04:16:59.815 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.816 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.816 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.816 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.905948+0000", 2023-11-01T04:16:59.816 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5600000000000004e-07 2023-11-01T04:16:59.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.817 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.817 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.817 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.905952+0000", 2023-11-01T04:16:59.817 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6750000000000001e-06 2023-11-01T04:16:59.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.817 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.818 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.818 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.905965+0000", 2023-11-01T04:16:59.818 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2655e-05 2023-11-01T04:16:59.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.818 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.818 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.819 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.905972+0000", 2023-11-01T04:16:59.819 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6000000000000003e-06 2023-11-01T04:16:59.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.819 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.819 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:16:59.819 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.106189+0000", 2023-11-01T04:16:59.820 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.200217426 2023-11-01T04:16:59.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.820 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.820 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:16:59.820 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.106225+0000", 2023-11-01T04:16:59.820 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5531000000000003e-05 2023-11-01T04:16:59.821 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.821 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.821 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.821 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.821 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.822 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:59.823 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-01T04:16:59.824 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:16:59.817+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:16:59.859 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-01T04:16:59.867 INFO:tasks.rados.rados.0.smithi190.stdout:274: done (7 left) 2023-11-01T04:16:59.868 INFO:tasks.rados.rados.0.smithi190.stdout:275: read oid 42 snap -1 2023-11-01T04:16:59.868 INFO:tasks.rados.rados.0.smithi190.stdout:275: expect (ObjNum 71 snap 8 seq_num 71) 2023-11-01T04:16:59.868 INFO:tasks.rados.rados.0.smithi190.stdout:276: delete oid 44 current snap is 24 2023-11-01T04:16:59.868 INFO:tasks.rados.rados.0.smithi190.stdout:268: expect (ObjNum 37 snap 0 seq_num 37) 2023-11-01T04:16:59.888 INFO:tasks.rados.rados.0.smithi190.stdout:270: expect (ObjNum 89 snap 20 seq_num 89) 2023-11-01T04:16:59.923 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:16:59.921+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:16:59.948 INFO:tasks.rados.rados.0.smithi190.stdout:269: expect (ObjNum 3 snap 0 seq_num 3) 2023-11-01T04:16:59.949 INFO:tasks.rados.rados.0.smithi190.stdout:264: finishing write tid 3 to smithi19039307-46 2023-11-01T04:16:59.950 INFO:tasks.rados.rados.0.smithi190.stdout:264: finishing write tid 4 to smithi19039307-46 2023-11-01T04:16:59.950 INFO:tasks.rados.rados.0.smithi190.stdout:264: finishing write tid 5 to smithi19039307-46 2023-11-01T04:16:59.958 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:16:59.958 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:16:59.958 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.959 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.bs5 79/31 ECSubWrite(tid=122, reqid=osd.10.0:122, at_version=79'46, trim_to=0'0, roll_forward_to=78'44))", 2023-11-01T04:16:59.959 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:59.883466+0000", 2023-11-01T04:16:59.959 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.075777623000000002, 2023-11-01T04:16:59.959 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.075801874000000005, 2023-11-01T04:16:59.959 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.960 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "queued for pg", 2023-11-01T04:16:59.960 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.960 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.960 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.960 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.883466+0000", 2023-11-01T04:16:59.961 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.961 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.961 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.961 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.962 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.883466+0000", 2023-11-01T04:16:59.962 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.962 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.962 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.962 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.963 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.883467+0000", 2023-11-01T04:16:59.963 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.3500000000000002e-07 2023-11-01T04:16:59.963 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.963 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.963 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.963 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.883518+0000", 2023-11-01T04:16:59.964 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1072000000000002e-05 2023-11-01T04:16:59.964 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.964 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.964 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.964 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.883536+0000", 2023-11-01T04:16:59.964 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8521000000000001e-05 2023-11-01T04:16:59.965 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.965 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.965 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:16:59.965 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.883551+0000", 2023-11-01T04:16:59.965 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5051999999999999e-05 2023-11-01T04:16:59.966 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.966 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.966 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.966 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.966 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.967 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.bs5 79/31 ECSubWrite(tid=123, reqid=osd.10.0:123, at_version=79'48, trim_to=0'0, roll_forward_to=78'44))", 2023-11-01T04:16:59.967 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:59.883582+0000", 2023-11-01T04:16:59.967 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.075661573999999995, 2023-11-01T04:16:59.967 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.075717393999999993, 2023-11-01T04:16:59.967 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:16:59.968 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "no flag points reached", 2023-11-01T04:16:59.968 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:16:59.968 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.968 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:16:59.968 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.883582+0000", 2023-11-01T04:16:59.968 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.969 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.969 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.969 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:16:59.969 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.883582+0000", 2023-11-01T04:16:59.969 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:16:59.970 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.970 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.970 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:16:59.970 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.883583+0000", 2023-11-01T04:16:59.970 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6800000000000001e-07 2023-11-01T04:16:59.970 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.971 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.971 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:16:59.971 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.883618+0000", 2023-11-01T04:16:59.971 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4783999999999998e-05 2023-11-01T04:16:59.971 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:16:59.972 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:16:59.972 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:16:59.972 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.883632+0000", 2023-11-01T04:16:59.972 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4455e-05 2023-11-01T04:16:59.972 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.972 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:16:59.973 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.973 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:16:59.973 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-11-01T04:16:59.973 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 2 2023-11-01T04:16:59.973 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:16:59.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-01T04:17:00.024 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:17:00.024+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:17:00.081 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 43 v 86 (ObjNum 84 snap 17 seq_num 84) dirty exists 2023-11-01T04:17:00.109 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:17:00.109 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:17:00.109 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2023-11-01T04:17:00.110 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2023-11-01T04:17:00.110 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:17:00.124 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:00.121+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:00.129 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-01T04:17:00.177 INFO:tasks.rados.rados.0.smithi190.stdout:266: done (8 left) 2023-11-01T04:17:00.178 INFO:tasks.rados.rados.0.smithi190.stdout:268: done (7 left) 2023-11-01T04:17:00.178 INFO:tasks.rados.rados.0.smithi190.stdout:269: done (6 left) 2023-11-01T04:17:00.178 INFO:tasks.rados.rados.0.smithi190.stdout:270: done (5 left) 2023-11-01T04:17:00.178 INFO:tasks.rados.rados.0.smithi190.stdout:276: done (4 left) 2023-11-01T04:17:00.179 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 8 current snap is 24 2023-11-01T04:17:00.179 INFO:tasks.rados.rados.0.smithi190.stdout:277: seq_num 108 ranges {294912=655360,950272=606208,1556480=475136,2031616=81920} 2023-11-01T04:17:00.194 INFO:tasks.rados.rados.0.smithi190.stdout:277: writing smithi19039307-8 from 294912 to 950272 tid 1 2023-11-01T04:17:00.204 INFO:tasks.rados.rados.0.smithi190.stdout:277: writing smithi19039307-8 from 950272 to 1556480 tid 2 2023-11-01T04:17:00.211 INFO:tasks.rados.rados.0.smithi190.stdout:277: writing smithi19039307-8 from 1556480 to 2031616 tid 3 2023-11-01T04:17:00.213 INFO:tasks.rados.rados.0.smithi190.stdout:277: writing smithi19039307-8 from 2031616 to 2113536 tid 4 2023-11-01T04:17:00.213 INFO:tasks.rados.rados.0.smithi190.stdout:278: rmattr oid 14 current snap is 24 2023-11-01T04:17:00.213 INFO:tasks.rados.rados.0.smithi190.stdout:278: done (5 left) 2023-11-01T04:17:00.213 INFO:tasks.rados.rados.0.smithi190.stdout:279: snap_remove snap 21 2023-11-01T04:17:00.225 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:00.223+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:00.293 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:17:00.293 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2023-11-01T04:17:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2023-11-01T04:17:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:17:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.as1 36/31 ECSubWrite(tid=24, reqid=client.4559.0:140, at_version=36'4, trim_to=0'0, roll_forward_to=34'2))", 2023-11-01T04:17:00.294 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.199076+0000", 2023-11-01T04:17:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 45.090927751000002, 2023-11-01T04:17:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20133796800000001, 2023-11-01T04:17:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.295 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199076+0000", 2023-11-01T04:17:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.296 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199076+0000", 2023-11-01T04:17:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.297 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199078+0000", 2023-11-01T04:17:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.623e-06 2023-11-01T04:17:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.298 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.299 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.299 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199121+0000", 2023-11-01T04:17:00.299 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3727999999999999e-05 2023-11-01T04:17:00.299 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.299 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.300 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.300 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.199278+0000", 2023-11-01T04:17:00.300 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015602899999999999 2023-11-01T04:17:00.300 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.300 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.300 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.301 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.299400+0000", 2023-11-01T04:17:00.301 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100122192 2023-11-01T04:17:00.301 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.301 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.301 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.302 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399577+0000", 2023-11-01T04:17:00.302 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10017696500000001 2023-11-01T04:17:00.302 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.302 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.302 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:00.303 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.399597+0000", 2023-11-01T04:17:00.303 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0213000000000001e-05 2023-11-01T04:17:00.303 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.303 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.303 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:00.304 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.400376+0000", 2023-11-01T04:17:00.304 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00077926799999999995 2023-11-01T04:17:00.304 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.304 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.304 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.304 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.400414+0000", 2023-11-01T04:17:00.305 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.7950000000000001e-05 2023-11-01T04:17:00.305 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.305 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.305 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.305 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.306 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.306 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds1 36/31 ECSubWrite(tid=30, reqid=client.4559.0:143, at_version=36'14, trim_to=0'0, roll_forward_to=36'13))", 2023-11-01T04:17:00.306 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.668264+0000", 2023-11-01T04:17:00.306 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 44.621739761000001, 2023-11-01T04:17:00.306 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20128238700000001, 2023-11-01T04:17:00.307 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.307 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:00.307 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.307 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.307 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.308 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.668264+0000", 2023-11-01T04:17:00.308 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.308 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.308 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.308 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.309 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.668264+0000", 2023-11-01T04:17:00.309 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.309 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.309 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.309 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.309 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.668265+0000", 2023-11-01T04:17:00.310 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0699999999999996e-07 2023-11-01T04:17:00.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.310 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.310 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.310 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.668318+0000", 2023-11-01T04:17:00.311 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3316000000000002e-05 2023-11-01T04:17:00.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.311 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.311 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.311 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.668335+0000", 2023-11-01T04:17:00.312 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6935999999999998e-05 2023-11-01T04:17:00.312 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.312 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.312 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.312 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.768415+0000", 2023-11-01T04:17:00.313 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100080154 2023-11-01T04:17:00.313 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.313 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.313 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.313 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.768508+0000", 2023-11-01T04:17:00.313 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2936000000000006e-05 2023-11-01T04:17:00.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.314 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.314 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:00.314 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.768521+0000", 2023-11-01T04:17:00.314 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2683999999999999e-05 2023-11-01T04:17:00.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.315 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.315 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:00.315 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.869474+0000", 2023-11-01T04:17:00.315 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100952606 2023-11-01T04:17:00.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.316 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.316 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.316 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.869547+0000", 2023-11-01T04:17:00.316 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2947999999999993e-05 2023-11-01T04:17:00.316 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.316 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.317 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.317 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.317 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/31 ECSubWrite(tid=236, reqid=client.4559.0:330, at_version=46'36, trim_to=0'0, roll_forward_to=45'33))", 2023-11-01T04:17:00.317 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.282825+0000", 2023-11-01T04:17:00.318 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 34.007178924000002, 2023-11-01T04:17:00.318 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20525125999999999, 2023-11-01T04:17:00.318 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.318 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:00.318 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.319 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.319 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.319 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282825+0000", 2023-11-01T04:17:00.319 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.319 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.320 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.320 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282825+0000", 2023-11-01T04:17:00.320 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.320 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.321 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.321 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282827+0000", 2023-11-01T04:17:00.321 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2800000000000002e-06 2023-11-01T04:17:00.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.321 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.322 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.322 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282955+0000", 2023-11-01T04:17:00.322 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012810400000000001 2023-11-01T04:17:00.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.322 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.323 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.323 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.283001+0000", 2023-11-01T04:17:00.323 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5411999999999998e-05 2023-11-01T04:17:00.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.323 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.323 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.324 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383146+0000", 2023-11-01T04:17:00.324 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100144996 2023-11-01T04:17:00.324 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.324 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.324 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383332+0000", 2023-11-01T04:17:00.325 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00018672099999999999 2023-11-01T04:17:00.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.325 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.325 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:00.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383371+0000", 2023-11-01T04:17:00.326 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8532999999999999e-05 2023-11-01T04:17:00.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.326 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.326 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:00.326 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.488032+0000", 2023-11-01T04:17:00.326 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104661192 2023-11-01T04:17:00.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.327 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.327 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.327 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.488076+0000", 2023-11-01T04:17:00.327 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4022e-05 2023-11-01T04:17:00.328 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.328 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.328 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.328 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.9s4 58/31 ECSubWrite(tid=63, reqid=osd.7.0:63, at_version=58'57, trim_to=0'0, roll_forward_to=57'53))", 2023-11-01T04:17:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:38.349078+0000", 2023-11-01T04:17:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 21.940925678999999, 2023-11-01T04:17:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20288057700000001, 2023-11-01T04:17:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.329 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.349078+0000", 2023-11-01T04:17:00.330 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.331 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.349078+0000", 2023-11-01T04:17:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.349080+0000", 2023-11-01T04:17:00.332 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.53e-06 2023-11-01T04:17:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.349136+0000", 2023-11-01T04:17:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5844999999999998e-05 2023-11-01T04:17:00.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.349166+0000", 2023-11-01T04:17:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0006999999999999e-05 2023-11-01T04:17:00.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.449262+0000", 2023-11-01T04:17:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10009615400000001 2023-11-01T04:17:00.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.549470+0000", 2023-11-01T04:17:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100208089 2023-11-01T04:17:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.336 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.549485+0000", 2023-11-01T04:17:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5150000000000001e-05 2023-11-01T04:17:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.337 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.551854+0000", 2023-11-01T04:17:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0023693910000000002 2023-11-01T04:17:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.338 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.551959+0000", 2023-11-01T04:17:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000104411 2023-11-01T04:17:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.es3 67/31 ECSubWrite(tid=77, reqid=client.4559.0:537, at_version=67'58, trim_to=0'0, roll_forward_to=67'56))", 2023-11-01T04:17:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:47.706034+0000", 2023-11-01T04:17:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 12.583969953, 2023-11-01T04:17:00.340 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20207028499999999, 2023-11-01T04:17:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.341 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.706034+0000", 2023-11-01T04:17:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.342 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.706034+0000", 2023-11-01T04:17:00.343 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.343 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.343 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.706036+0000", 2023-11-01T04:17:00.343 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6670000000000001e-06 2023-11-01T04:17:00.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.344 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.344 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.344 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.706723+0000", 2023-11-01T04:17:00.344 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00068749900000000005 2023-11-01T04:17:00.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.345 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.345 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.706754+0000", 2023-11-01T04:17:00.345 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0985999999999998e-05 2023-11-01T04:17:00.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.346 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.346 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.706779+0000", 2023-11-01T04:17:00.346 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4752999999999998e-05 2023-11-01T04:17:00.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.346 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.347 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.347 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.806964+0000", 2023-11-01T04:17:00.347 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10018563699999999 2023-11-01T04:17:00.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.347 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:00.348 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.806974+0000", 2023-11-01T04:17:00.348 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2660000000000007e-06 2023-11-01T04:17:00.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.348 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.348 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:00.348 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.908070+0000", 2023-11-01T04:17:00.349 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101096143 2023-11-01T04:17:00.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.349 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.349 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.908104+0000", 2023-11-01T04:17:00.350 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4334e-05 2023-11-01T04:17:00.350 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.350 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.350 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.351 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:536 3.as1 3:54622f20:::smithi19039307-19:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:00.351 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:47.776853+0000", 2023-11-01T04:17:00.351 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 12.513151082, 2023-11-01T04:17:00.351 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6173196650000001, 2023-11-01T04:17:00.351 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.352 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:00.352 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:00.352 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:00.352 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:00.352 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 536 2023-11-01T04:17:00.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.353 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.353 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.353 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.353 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.776853+0000", 2023-11-01T04:17:00.353 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.354 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.354 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.354 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.776853+0000", 2023-11-01T04:17:00.354 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.355 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.355 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.776855+0000", 2023-11-01T04:17:00.355 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.463e-06 2023-11-01T04:17:00.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.355 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.356 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.356 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.776864+0000", 2023-11-01T04:17:00.356 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5529999999999993e-06 2023-11-01T04:17:00.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.356 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.357 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.357 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.776886+0000", 2023-11-01T04:17:00.357 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2447e-05 2023-11-01T04:17:00.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.358 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.358 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.358 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.776903+0000", 2023-11-01T04:17:00.358 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6609e-05 2023-11-01T04:17:00.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.358 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.359 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.359 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.777058+0000", 2023-11-01T04:17:00.359 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015479799999999999 2023-11-01T04:17:00.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.359 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.360 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2023-11-01T04:17:00.360 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.777071+0000", 2023-11-01T04:17:00.360 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2965999999999999e-05 2023-11-01T04:17:00.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.360 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.361 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.187223+0000", 2023-11-01T04:17:00.361 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.41015271599999997 2023-11-01T04:17:00.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.361 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.361 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2023-11-01T04:17:00.362 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.187230+0000", 2023-11-01T04:17:00.362 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.046e-06 2023-11-01T04:17:00.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.362 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.362 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.191371+0000", 2023-11-01T04:17:00.363 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.004140748 2023-11-01T04:17:00.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.363 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:00.363 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.191657+0000", 2023-11-01T04:17:00.363 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00028538899999999999 2023-11-01T04:17:00.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.364 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.364 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.394172+0000", 2023-11-01T04:17:00.364 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20251593000000001 2023-11-01T04:17:00.365 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.365 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.365 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.365 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.365 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s3 68/31 ECSubWrite(tid=314, reqid=osd.13.0:314, at_version=68'70, trim_to=0'0, roll_forward_to=67'68))", 2023-11-01T04:17:00.366 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:47.785888+0000", 2023-11-01T04:17:00.366 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 12.504115629999999, 2023-11-01T04:17:00.366 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.300970022, 2023-11-01T04:17:00.366 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.366 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:00.366 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.367 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.367 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.785888+0000", 2023-11-01T04:17:00.367 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.368 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.368 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.368 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.785888+0000", 2023-11-01T04:17:00.368 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.368 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.369 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.369 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.369 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.785889+0000", 2023-11-01T04:17:00.369 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8500000000000002e-07 2023-11-01T04:17:00.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.369 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.370 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.370 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.785916+0000", 2023-11-01T04:17:00.370 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7180999999999999e-05 2023-11-01T04:17:00.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.370 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.371 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.371 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.785938+0000", 2023-11-01T04:17:00.371 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1727999999999999e-05 2023-11-01T04:17:00.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.371 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.372 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.785949+0000", 2023-11-01T04:17:00.372 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1328000000000001e-05 2023-11-01T04:17:00.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.373 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.986492+0000", 2023-11-01T04:17:00.373 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20054274699999999 2023-11-01T04:17:00.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.374 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:00.374 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.986502+0000", 2023-11-01T04:17:00.374 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0876000000000001e-05 2023-11-01T04:17:00.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.375 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:00.375 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.086786+0000", 2023-11-01T04:17:00.375 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10028398099999999 2023-11-01T04:17:00.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.376 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.376 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.376 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.086858+0000", 2023-11-01T04:17:00.376 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1796000000000002e-05 2023-11-01T04:17:00.376 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.377 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.377 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.377 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s3 68/31 ECSubWrite(tid=315, reqid=osd.13.0:315, at_version=68'72, trim_to=0'0, roll_forward_to=67'68))", 2023-11-01T04:17:00.377 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:47.788107+0000", 2023-11-01T04:17:00.378 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 12.501896742, 2023-11-01T04:17:00.378 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.39927050600000003, 2023-11-01T04:17:00.378 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.378 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:00.378 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.379 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.379 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.788107+0000", 2023-11-01T04:17:00.379 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.379 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.380 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.380 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.788107+0000", 2023-11-01T04:17:00.380 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.380 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.381 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.381 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.788108+0000", 2023-11-01T04:17:00.381 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0699999999999997e-07 2023-11-01T04:17:00.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.381 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.382 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.382 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.788146+0000", 2023-11-01T04:17:00.382 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8209e-05 2023-11-01T04:17:00.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.382 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.383 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.383 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.788163+0000", 2023-11-01T04:17:00.383 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6900000000000001e-05 2023-11-01T04:17:00.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.383 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.383 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.384 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.788176+0000", 2023-11-01T04:17:00.384 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.272e-05 2023-11-01T04:17:00.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.384 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.384 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.385 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.086965+0000", 2023-11-01T04:17:00.385 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.29878976600000001 2023-11-01T04:17:00.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.385 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.385 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:00.385 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.086990+0000", 2023-11-01T04:17:00.386 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4457e-05 2023-11-01T04:17:00.386 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.386 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.386 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:00.386 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.187306+0000", 2023-11-01T04:17:00.386 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100315998 2023-11-01T04:17:00.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.387 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.387 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.387 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.187378+0000", 2023-11-01T04:17:00.387 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1848999999999996e-05 2023-11-01T04:17:00.387 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.388 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.388 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.388 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.388 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.388 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s3 69/31 ECSubWrite(tid=323, reqid=client.4559.0:568, at_version=69'77, trim_to=0'0, roll_forward_to=68'74))", 2023-11-01T04:17:00.388 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:49.614285+0000", 2023-11-01T04:17:00.389 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 10.675718395000001, 2023-11-01T04:17:00.389 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.205251824, 2023-11-01T04:17:00.389 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.389 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:00.389 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.390 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.390 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.390 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.614285+0000", 2023-11-01T04:17:00.390 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.390 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.390 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.391 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.391 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.614285+0000", 2023-11-01T04:17:00.391 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.391 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.392 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.392 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.392 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.614287+0000", 2023-11-01T04:17:00.392 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1230000000000001e-06 2023-11-01T04:17:00.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.392 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.393 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.393 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.614565+0000", 2023-11-01T04:17:00.393 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00027808299999999999 2023-11-01T04:17:00.393 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.393 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.393 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.394 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.614600+0000", 2023-11-01T04:17:00.394 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5719000000000001e-05 2023-11-01T04:17:00.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.394 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.394 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.395 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.714696+0000", 2023-11-01T04:17:00.395 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100095195 2023-11-01T04:17:00.395 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.395 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.395 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.396 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.714866+0000", 2023-11-01T04:17:00.396 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00017038099999999999 2023-11-01T04:17:00.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.396 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.396 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:00.397 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.714882+0000", 2023-11-01T04:17:00.397 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5593000000000001e-05 2023-11-01T04:17:00.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.397 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.397 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:00.398 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.819453+0000", 2023-11-01T04:17:00.398 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104571505 2023-11-01T04:17:00.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.398 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.398 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.399 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.819537+0000", 2023-11-01T04:17:00.399 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4225000000000001e-05 2023-11-01T04:17:00.399 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.399 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.399 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.400 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:580 3.ds1 3:b59be519:::smithi19039307-27:head [append 0~0] snapc 22=[22,1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2023-11-01T04:17:00.400 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:50.129017+0000", 2023-11-01T04:17:00.400 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 10.160986929, 2023-11-01T04:17:00.401 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20754502599999999, 2023-11-01T04:17:00.401 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.401 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:00.401 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:00.401 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:00.401 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:00.402 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 580 2023-11-01T04:17:00.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.402 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.402 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.402 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.403 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.129017+0000", 2023-11-01T04:17:00.403 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.403 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.403 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.403 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.129017+0000", 2023-11-01T04:17:00.404 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.404 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.404 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.404 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.129019+0000", 2023-11-01T04:17:00.404 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4269999999999998e-06 2023-11-01T04:17:00.405 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.405 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.405 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.405 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.129030+0000", 2023-11-01T04:17:00.405 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0746e-05 2023-11-01T04:17:00.406 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.406 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.406 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.406 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.129062+0000", 2023-11-01T04:17:00.406 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2276000000000001e-05 2023-11-01T04:17:00.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.407 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.407 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.407 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.129078+0000", 2023-11-01T04:17:00.407 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5886999999999999e-05 2023-11-01T04:17:00.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.408 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.408 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.408 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.129195+0000", 2023-11-01T04:17:00.408 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000116394 2023-11-01T04:17:00.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.409 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.409 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:00.409 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.129481+0000", 2023-11-01T04:17:00.409 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00028630000000000002 2023-11-01T04:17:00.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.410 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.410 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:00.410 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.130623+0000", 2023-11-01T04:17:00.410 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001142386 2023-11-01T04:17:00.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.410 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.411 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:00.411 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.135088+0000", 2023-11-01T04:17:00.411 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0044649920000000001 2023-11-01T04:17:00.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.411 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.412 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:00.412 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.336260+0000", 2023-11-01T04:17:00.412 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20117206500000001 2023-11-01T04:17:00.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.412 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.412 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.413 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.336562+0000", 2023-11-01T04:17:00.413 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00030155299999999998 2023-11-01T04:17:00.413 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.413 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.413 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.414 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.414 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.bs5 72/31 ECSubWrite(tid=100, reqid=osd.10.0:97, at_version=0'0, trim_to=0'0, roll_forward_to=72'28))", 2023-11-01T04:17:00.414 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:53.026595+0000", 2023-11-01T04:17:00.414 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 7.2634093650000002, 2023-11-01T04:17:00.414 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.201262777, 2023-11-01T04:17:00.415 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.415 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:00.415 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.415 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.415 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.415 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.026595+0000", 2023-11-01T04:17:00.416 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.416 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.416 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.416 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.026595+0000", 2023-11-01T04:17:00.416 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.417 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.417 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.417 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.026596+0000", 2023-11-01T04:17:00.417 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3829999999999999e-06 2023-11-01T04:17:00.418 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.418 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.418 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.418 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.026610+0000", 2023-11-01T04:17:00.418 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4168999999999999e-05 2023-11-01T04:17:00.418 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.419 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.419 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.419 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.026629+0000", 2023-11-01T04:17:00.419 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8453000000000001e-05 2023-11-01T04:17:00.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.419 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.420 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.420 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.126779+0000", 2023-11-01T04:17:00.420 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100150423 2023-11-01T04:17:00.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.420 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.421 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.421 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.126953+0000", 2023-11-01T04:17:00.421 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00017453600000000001 2023-11-01T04:17:00.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.421 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.421 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:00.422 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.126978+0000", 2023-11-01T04:17:00.422 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4454999999999999e-05 2023-11-01T04:17:00.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.422 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.422 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:00.423 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.227757+0000", 2023-11-01T04:17:00.423 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100778676 2023-11-01T04:17:00.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.423 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.423 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.423 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.227857+0000", 2023-11-01T04:17:00.424 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000100682 2023-11-01T04:17:00.424 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.424 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.424 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.424 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.425 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:619 3.ds1 3:ba00e0a8:::smithi19039307-23:head [append 0~589824] snapc 27=[27,26,22,1b,19,18,12,11] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2023-11-01T04:17:00.425 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:54.774547+0000", 2023-11-01T04:17:00.425 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.515457176, 2023-11-01T04:17:00.425 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.32359992700000001, 2023-11-01T04:17:00.425 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.426 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:00.426 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:00.426 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:00.426 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:00.426 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 619 2023-11-01T04:17:00.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.427 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.427 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.427 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.427 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.774547+0000", 2023-11-01T04:17:00.427 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.428 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.428 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.428 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.774547+0000", 2023-11-01T04:17:00.428 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.429 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.429 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.429 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.774550+0000", 2023-11-01T04:17:00.429 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8399999999999999e-06 2023-11-01T04:17:00.429 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.430 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.430 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.430 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.775642+0000", 2023-11-01T04:17:00.430 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0010924999999999999 2023-11-01T04:17:00.430 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.431 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.431 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.431 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.775670+0000", 2023-11-01T04:17:00.431 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7517000000000001e-05 2023-11-01T04:17:00.431 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.432 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.432 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.432 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.775684+0000", 2023-11-01T04:17:00.432 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4112e-05 2023-11-01T04:17:00.432 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.432 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.433 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.433 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.782384+0000", 2023-11-01T04:17:00.433 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0067004439999999998 2023-11-01T04:17:00.433 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.433 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.434 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:00.434 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.782503+0000", 2023-11-01T04:17:00.434 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000118773 2023-11-01T04:17:00.434 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.434 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.434 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:00.435 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.783538+0000", 2023-11-01T04:17:00.435 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001035338 2023-11-01T04:17:00.435 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.435 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.435 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:00.436 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.785653+0000", 2023-11-01T04:17:00.436 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002114997 2023-11-01T04:17:00.436 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.436 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.436 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:00.436 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.097825+0000", 2023-11-01T04:17:00.437 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.31217141999999998 2023-11-01T04:17:00.437 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.437 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.437 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.437 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.098147+0000", 2023-11-01T04:17:00.438 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00032198600000000002 2023-11-01T04:17:00.438 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.438 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.438 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.438 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.439 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.439 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:620 3.ds1 3:ba00e0a8:::smithi19039307-23:head [append 0~540672] snapc 27=[27,26,22,1b,19,18,12,11] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2023-11-01T04:17:00.439 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:54.781704+0000", 2023-11-01T04:17:00.439 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.5083003279999998, 2023-11-01T04:17:00.439 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.31728704400000002, 2023-11-01T04:17:00.440 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.440 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:00.440 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:00.440 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:00.440 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:00.440 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 620 2023-11-01T04:17:00.441 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.441 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.441 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.441 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.441 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.781704+0000", 2023-11-01T04:17:00.442 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.442 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.442 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.442 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.442 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.781704+0000", 2023-11-01T04:17:00.443 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.443 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.443 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.443 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.443 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.781706+0000", 2023-11-01T04:17:00.443 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4549999999999998e-06 2023-11-01T04:17:00.444 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.444 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.444 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.444 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.782222+0000", 2023-11-01T04:17:00.444 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00051576599999999997 2023-11-01T04:17:00.444 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.445 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.445 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.445 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.782250+0000", 2023-11-01T04:17:00.445 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7750999999999999e-05 2023-11-01T04:17:00.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.445 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.446 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.446 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.782267+0000", 2023-11-01T04:17:00.446 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7716000000000001e-05 2023-11-01T04:17:00.446 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.446 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.447 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.447 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.785897+0000", 2023-11-01T04:17:00.447 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0036297730000000002 2023-11-01T04:17:00.447 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.447 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.447 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:00.448 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.786001+0000", 2023-11-01T04:17:00.448 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010424099999999999 2023-11-01T04:17:00.448 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.448 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.448 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:00.448 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.786754+0000", 2023-11-01T04:17:00.449 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000753091 2023-11-01T04:17:00.449 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.449 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.449 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:00.449 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.788391+0000", 2023-11-01T04:17:00.450 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001636289 2023-11-01T04:17:00.450 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.450 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.450 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:00.450 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.098663+0000", 2023-11-01T04:17:00.450 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.31027191900000001 2023-11-01T04:17:00.451 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.451 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.451 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.451 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.098991+0000", 2023-11-01T04:17:00.451 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00032804299999999998 2023-11-01T04:17:00.452 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.452 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.452 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.452 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.452 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.452 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:621 3.ds1 3:ba00e0a8:::smithi19039307-23:head [append 0~622592] snapc 27=[27,26,22,1b,19,18,12,11] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2023-11-01T04:17:00.453 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:54.789814+0000", 2023-11-01T04:17:00.453 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.5001902339999997, 2023-11-01T04:17:00.453 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30989988200000002, 2023-11-01T04:17:00.453 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.453 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:00.453 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:00.454 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:00.454 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:00.454 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 621 2023-11-01T04:17:00.454 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.454 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.454 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.455 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.455 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.789814+0000", 2023-11-01T04:17:00.455 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.455 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.455 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.456 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.456 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.789814+0000", 2023-11-01T04:17:00.456 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.456 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.456 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.456 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.457 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.789815+0000", 2023-11-01T04:17:00.457 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.556e-06 2023-11-01T04:17:00.457 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.457 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.457 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.457 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790468+0000", 2023-11-01T04:17:00.458 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000652453 2023-11-01T04:17:00.458 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.458 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.458 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.458 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790486+0000", 2023-11-01T04:17:00.458 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7861000000000001e-05 2023-11-01T04:17:00.459 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.459 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.459 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.459 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790494+0000", 2023-11-01T04:17:00.459 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5450000000000003e-06 2023-11-01T04:17:00.459 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.460 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.460 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.460 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.890901+0000", 2023-11-01T04:17:00.460 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100406963 2023-11-01T04:17:00.460 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.461 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.461 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:00.461 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.891217+0000", 2023-11-01T04:17:00.461 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00031643900000000001 2023-11-01T04:17:00.461 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.461 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.462 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:00.462 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.893720+0000", 2023-11-01T04:17:00.462 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0025027389999999999 2023-11-01T04:17:00.462 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.462 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.463 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:00.463 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.896858+0000", 2023-11-01T04:17:00.463 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0031375919999999998 2023-11-01T04:17:00.463 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.463 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.463 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:00.464 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.099333+0000", 2023-11-01T04:17:00.464 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20247476 2023-11-01T04:17:00.464 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.464 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.464 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.464 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.099714+0000", 2023-11-01T04:17:00.465 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00038097400000000002 2023-11-01T04:17:00.465 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.465 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.465 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.465 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.465 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.466 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:622 3.ds1 3:ba00e0a8:::smithi19039307-23:head [setxattr _header (57) in=7b] snapc 27=[27,26,22,1b,19,18,12,11] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2023-11-01T04:17:00.466 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:54.790514+0000", 2023-11-01T04:17:00.466 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.4994898760000002, 2023-11-01T04:17:00.466 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.51671497700000002, 2023-11-01T04:17:00.466 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.467 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:00.467 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:00.467 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:00.467 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:00.467 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 622 2023-11-01T04:17:00.467 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.468 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.468 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.468 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.468 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790514+0000", 2023-11-01T04:17:00.468 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.469 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.469 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.469 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.469 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790514+0000", 2023-11-01T04:17:00.469 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.470 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.470 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.470 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.470 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790515+0000", 2023-11-01T04:17:00.470 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.133e-06 2023-11-01T04:17:00.471 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.471 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.471 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.471 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790518+0000", 2023-11-01T04:17:00.471 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.123e-06 2023-11-01T04:17:00.472 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.472 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.472 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.472 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790527+0000", 2023-11-01T04:17:00.472 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8449999999999992e-06 2023-11-01T04:17:00.473 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.473 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.473 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.473 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790534+0000", 2023-11-01T04:17:00.473 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8739999999999997e-06 2023-11-01T04:17:00.473 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.474 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.474 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.474 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.100566+0000", 2023-11-01T04:17:00.474 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.31003165700000002 2023-11-01T04:17:00.474 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.475 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.475 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:00.475 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.100855+0000", 2023-11-01T04:17:00.475 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000289074 2023-11-01T04:17:00.475 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.476 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.476 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:00.476 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.102192+0000", 2023-11-01T04:17:00.476 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001336814 2023-11-01T04:17:00.476 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.476 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.477 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:00.477 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.104907+0000", 2023-11-01T04:17:00.477 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0027159020000000001 2023-11-01T04:17:00.477 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.477 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.477 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:00.478 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.306899+0000", 2023-11-01T04:17:00.478 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20199112599999999 2023-11-01T04:17:00.478 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.478 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.478 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.479 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.307229+0000", 2023-11-01T04:17:00.479 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00033042899999999998 2023-11-01T04:17:00.479 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.479 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.479 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.479 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.480 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.480 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:623 3.ds1 3:ba00e0a8:::smithi19039307-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2023-11-01T04:17:00.480 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:54.790541+0000", 2023-11-01T04:17:00.480 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.4994629079999999, 2023-11-01T04:17:00.480 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.71895732199999995, 2023-11-01T04:17:00.480 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.481 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:00.481 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:00.481 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:00.481 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:00.481 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 623 2023-11-01T04:17:00.482 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.482 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.482 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.482 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.482 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790541+0000", 2023-11-01T04:17:00.482 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.483 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.483 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.483 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.483 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790541+0000", 2023-11-01T04:17:00.483 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.483 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.484 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.484 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.484 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790541+0000", 2023-11-01T04:17:00.484 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.58e-07 2023-11-01T04:17:00.484 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.485 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.485 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.485 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790544+0000", 2023-11-01T04:17:00.485 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8420000000000001e-06 2023-11-01T04:17:00.485 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.485 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.486 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.486 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790552+0000", 2023-11-01T04:17:00.486 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4789999999999999e-06 2023-11-01T04:17:00.486 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.486 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.486 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.487 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790556+0000", 2023-11-01T04:17:00.487 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5920000000000002e-06 2023-11-01T04:17:00.487 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.487 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.487 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.488 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.102797+0000", 2023-11-01T04:17:00.488 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.31224017199999998 2023-11-01T04:17:00.488 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.488 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.488 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:00.488 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.102961+0000", 2023-11-01T04:17:00.489 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016474999999999999 2023-11-01T04:17:00.489 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.489 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.489 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.489 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.407367+0000", 2023-11-01T04:17:00.489 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30440549900000002 2023-11-01T04:17:00.490 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.490 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.490 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:00.490 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.407516+0000", 2023-11-01T04:17:00.490 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014955400000000001 2023-11-01T04:17:00.490 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.491 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.491 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.491 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.509498+0000", 2023-11-01T04:17:00.491 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101981976 2023-11-01T04:17:00.491 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.492 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.492 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.492 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.492 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.492 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.ds1 74/68 ECSubWriteReply(tid=12, last_complete=74'85, committed=1, applied=1))", 2023-11-01T04:17:00.492 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:54.896818+0000", 2023-11-01T04:17:00.493 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.3931857020000002, 2023-11-01T04:17:00.493 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20155440699999999, 2023-11-01T04:17:00.493 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.493 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:00.493 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.493 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.494 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.494 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.896818+0000", 2023-11-01T04:17:00.494 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.494 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.494 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.494 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.495 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.896818+0000", 2023-11-01T04:17:00.495 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.495 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.495 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.495 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.495 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.896819+0000", 2023-11-01T04:17:00.496 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.113e-06 2023-11-01T04:17:00.496 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.496 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.496 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.496 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.896826+0000", 2023-11-01T04:17:00.497 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6900000000000003e-06 2023-11-01T04:17:00.497 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.497 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.497 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.497 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.896843+0000", 2023-11-01T04:17:00.497 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7348999999999999e-05 2023-11-01T04:17:00.498 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.498 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.498 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.498 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.896857+0000", 2023-11-01T04:17:00.498 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3923999999999999e-05 2023-11-01T04:17:00.498 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.499 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.499 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.499 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.098332+0000", 2023-11-01T04:17:00.499 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20147490900000001 2023-11-01T04:17:00.499 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.500 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.500 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.500 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.098373+0000", 2023-11-01T04:17:00.500 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0422e-05 2023-11-01T04:17:00.500 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.500 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.501 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.501 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.501 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.501 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDPGPush(3.3s5 74/72 [PushOp(3:ca2966a4:::smithi19039307-37:1e, version: 67'49, data_included: [0~151552], data_size: 151552, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:ca2966a4:::smithi19039307-37:1e@67'49, size: 606208, copy_subset: [], clone_subset: {}, snapset: 1e=[]:{1e: [1e, 1c, 1b, 19, 18, 15, 12, 11, 4]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 606208, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2023-11-01T04:17:00.501 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:55.306605+0000", 2023-11-01T04:17:00.501 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 4.9833990220000004, 2023-11-01T04:17:00.502 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.202029189, 2023-11-01T04:17:00.502 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.502 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:00.502 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.502 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.503 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.503 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.306605+0000", 2023-11-01T04:17:00.503 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.503 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.503 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.503 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.504 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.306605+0000", 2023-11-01T04:17:00.504 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.504 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.504 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.504 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.504 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.306606+0000", 2023-11-01T04:17:00.505 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.217e-06 2023-11-01T04:17:00.505 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.505 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.505 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.505 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.306849+0000", 2023-11-01T04:17:00.506 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00024292300000000001 2023-11-01T04:17:00.506 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.506 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.506 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.506 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.306884+0000", 2023-11-01T04:17:00.507 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5135000000000002e-05 2023-11-01T04:17:00.507 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.507 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.507 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.507 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.407042+0000", 2023-11-01T04:17:00.507 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100158185 2023-11-01T04:17:00.508 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.508 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.508 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.508 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.507404+0000", 2023-11-01T04:17:00.508 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10036146999999999 2023-11-01T04:17:00.509 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.509 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.509 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.509 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.508634+0000", 2023-11-01T04:17:00.510 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001230259 2023-11-01T04:17:00.510 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.510 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.510 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.510 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.510 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.511 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.es3 74/73 ECSubWrite(tid=114, reqid=osd.11.0:114, at_version=74'100, trim_to=0'0, roll_forward_to=74'98))", 2023-11-01T04:17:00.511 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:55.508921+0000", 2023-11-01T04:17:00.511 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 4.781082509, 2023-11-01T04:17:00.511 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.202364931, 2023-11-01T04:17:00.511 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.512 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:00.512 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.512 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.512 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.512 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.508921+0000", 2023-11-01T04:17:00.512 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.513 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.513 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.513 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.513 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.508921+0000", 2023-11-01T04:17:00.513 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.514 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.514 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.514 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.514 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.508922+0000", 2023-11-01T04:17:00.514 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.13e-07 2023-11-01T04:17:00.514 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.515 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.515 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.515 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.508998+0000", 2023-11-01T04:17:00.515 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5894000000000002e-05 2023-11-01T04:17:00.515 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.516 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.516 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.516 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.509016+0000", 2023-11-01T04:17:00.516 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8156e-05 2023-11-01T04:17:00.516 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.517 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.517 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.517 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.609112+0000", 2023-11-01T04:17:00.517 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10009633 2023-11-01T04:17:00.517 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.517 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.518 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.518 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.609252+0000", 2023-11-01T04:17:00.518 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013997099999999999 2023-11-01T04:17:00.518 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.519 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.519 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:00.519 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.609273+0000", 2023-11-01T04:17:00.519 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0726000000000001e-05 2023-11-01T04:17:00.519 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.519 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.520 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:00.520 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.711195+0000", 2023-11-01T04:17:00.520 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10192179799999999 2023-11-01T04:17:00.520 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.520 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.521 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.521 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.711286+0000", 2023-11-01T04:17:00.521 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.1442999999999998e-05 2023-11-01T04:17:00.521 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.521 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.522 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.522 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.522 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.522 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.es3 78/73 ECSubWrite(tid=128, reqid=osd.11.0:128, at_version=78'118, trim_to=0'0, roll_forward_to=78'116))", 2023-11-01T04:17:00.522 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:59.086013+0000", 2023-11-01T04:17:00.522 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 1.2039905850000001, 2023-11-01T04:17:00.523 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20312366400000001, 2023-11-01T04:17:00.523 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:00.523 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:00.523 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:00.523 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.524 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:00.524 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.086013+0000", 2023-11-01T04:17:00.524 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.524 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.524 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:00.525 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.086013+0000", 2023-11-01T04:17:00.525 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:00.525 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.525 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.525 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:00.526 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.086015+0000", 2023-11-01T04:17:00.526 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8530000000000001e-06 2023-11-01T04:17:00.526 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.526 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.526 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:00.526 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.086058+0000", 2023-11-01T04:17:00.527 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2354999999999997e-05 2023-11-01T04:17:00.527 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.527 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.527 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:00.527 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.086090+0000", 2023-11-01T04:17:00.527 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2749000000000001e-05 2023-11-01T04:17:00.528 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.528 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.528 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:00.528 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.186237+0000", 2023-11-01T04:17:00.528 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100146376 2023-11-01T04:17:00.528 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.529 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.529 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:00.529 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.286508+0000", 2023-11-01T04:17:00.529 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100270991 2023-11-01T04:17:00.529 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.530 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.530 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:00.530 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.286519+0000", 2023-11-01T04:17:00.530 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1715e-05 2023-11-01T04:17:00.530 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.530 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.531 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:00.531 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.289097+0000", 2023-11-01T04:17:00.531 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0025772490000000002 2023-11-01T04:17:00.531 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:00.531 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:00.532 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:00.532 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.289137+0000", 2023-11-01T04:17:00.532 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0376000000000001e-05 2023-11-01T04:17:00.532 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.532 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.532 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.533 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:00.533 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:00.533 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:17:00.534 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-01T04:17:00.535 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:17:00.531+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:17:00.636 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:00.633+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:00.665 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:00.665 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:00.679 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-01T04:17:00.737 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:17:00.733+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:17:00.796 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:00.796 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:00.809 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-01T04:17:00.859 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:17:00.860+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:17:00.860 INFO:teuthology.orchestra.run.smithi153.stderr:noscrub is unset 2023-11-01T04:17:00.868 INFO:tasks.rados.rados.0.smithi190.stdout:279: done (5 left) 2023-11-01T04:17:00.868 INFO:tasks.rados.rados.0.smithi190.stdout:280: snap_remove snap 14 2023-11-01T04:17:00.929 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:00.930 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:00.938 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:17:00.936+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:17:00.944 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-01T04:17:01.038 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:17:01.035+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:17:01.093 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:01.097 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-01T04:17:01.140 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:01.136+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:01.199 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:01.201 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:01.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-01T04:17:01.248 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:17:01.248+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:17:01.305 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:01.308 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:01.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-01T04:17:01.357 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:01.356+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:01.440 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:01.440 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:17:01.441 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2023-11-01T04:17:01.441 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:01.441 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:01.439+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:01.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-01T04:17:01.542 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:17:01.537+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:17:01.621 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:01.621 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:17:01.621 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:17:01.621 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:17:01.621 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:01.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-01T04:17:01.642 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:17:01.640+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:17:01.743 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:01.740+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:01.763 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:01.763 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:17:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:17:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:17:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 36/31 ECSubWrite(tid=18, reqid=client.4559.0:125, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:17:01.764 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.281930+0000", 2023-11-01T04:17:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.478751965000001, 2023-11-01T04:17:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20154303100000001, 2023-11-01T04:17:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.765 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281930+0000", 2023-11-01T04:17:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.766 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281930+0000", 2023-11-01T04:17:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.767 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281931+0000", 2023-11-01T04:17:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6400000000000001e-07 2023-11-01T04:17:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.768 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281961+0000", 2023-11-01T04:17:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9621999999999999e-05 2023-11-01T04:17:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.769 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.281987+0000", 2023-11-01T04:17:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6316e-05 2023-11-01T04:17:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.770 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.382103+0000", 2023-11-01T04:17:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100116649 2023-11-01T04:17:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.771 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.482335+0000", 2023-11-01T04:17:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100231813 2023-11-01T04:17:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.482364+0000", 2023-11-01T04:17:01.772 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8829000000000001e-05 2023-11-01T04:17:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.483427+0000", 2023-11-01T04:17:01.773 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010627970000000001 2023-11-01T04:17:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.483473+0000", 2023-11-01T04:17:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6241e-05 2023-11-01T04:17:01.774 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.775 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.0s3 36/31 ECSubWrite(tid=4, reqid=client.4559.0:136, at_version=36'3, trim_to=0'0, roll_forward_to=33'2))", 2023-11-01T04:17:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.284706+0000", 2023-11-01T04:17:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.475976056999997, 2023-11-01T04:17:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202677317, 2023-11-01T04:17:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.776 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.777 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.777 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.777 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.284706+0000", 2023-11-01T04:17:01.777 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.778 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.778 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.778 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.284706+0000", 2023-11-01T04:17:01.778 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.779 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.779 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.779 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.284707+0000", 2023-11-01T04:17:01.779 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8599999999999998e-07 2023-11-01T04:17:01.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.780 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.780 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.780 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.285485+0000", 2023-11-01T04:17:01.780 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077885900000000002 2023-11-01T04:17:01.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.780 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.781 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.781 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.285511+0000", 2023-11-01T04:17:01.781 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5378999999999999e-05 2023-11-01T04:17:01.781 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.781 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.782 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.782 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385600+0000", 2023-11-01T04:17:01.782 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100089631 2023-11-01T04:17:01.782 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.782 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.783 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.783 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385749+0000", 2023-11-01T04:17:01.783 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014876399999999999 2023-11-01T04:17:01.783 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.783 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.783 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.784 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.385771+0000", 2023-11-01T04:17:01.784 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1535e-05 2023-11-01T04:17:01.784 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.784 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.784 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.785 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.487315+0000", 2023-11-01T04:17:01.785 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101544735 2023-11-01T04:17:01.785 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.785 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.785 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.785 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.487383+0000", 2023-11-01T04:17:01.786 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7928000000000004e-05 2023-11-01T04:17:01.786 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.786 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.786 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.786 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.787 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.787 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 36/31 ECSubWrite(tid=14, reqid=client.4559.0:146, at_version=36'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:17:01.787 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.322569+0000", 2023-11-01T04:17:01.787 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.438113162999997, 2023-11-01T04:17:01.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20180657299999999, 2023-11-01T04:17:01.787 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.788 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.788 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.788 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322569+0000", 2023-11-01T04:17:01.789 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.789 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.789 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322569+0000", 2023-11-01T04:17:01.790 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.790 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.790 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.790 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.790 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322571+0000", 2023-11-01T04:17:01.790 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8029999999999999e-06 2023-11-01T04:17:01.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.791 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.791 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.791 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322759+0000", 2023-11-01T04:17:01.791 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018862300000000001 2023-11-01T04:17:01.792 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.792 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322790+0000", 2023-11-01T04:17:01.792 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0712e-05 2023-11-01T04:17:01.793 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.793 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.793 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.793 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.322807+0000", 2023-11-01T04:17:01.793 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6566e-05 2023-11-01T04:17:01.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.794 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.794 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.794 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.423031+0000", 2023-11-01T04:17:01.794 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10022463600000001 2023-11-01T04:17:01.795 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.795 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.795 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.795 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.423046+0000", 2023-11-01T04:17:01.795 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4632e-05 2023-11-01T04:17:01.795 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.796 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.796 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.796 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524322+0000", 2023-11-01T04:17:01.796 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101275743 2023-11-01T04:17:01.796 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.797 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.797 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.797 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.524376+0000", 2023-11-01T04:17:01.797 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3857999999999998e-05 2023-11-01T04:17:01.797 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.798 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.798 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.798 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.798 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.798 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 37/31 ECSubWrite(tid=22, reqid=client.4559.0:197, at_version=37'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:17:01.799 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:17:01.799 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 44.720410551000001, 2023-11-01T04:17:01.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.301677274, 2023-11-01T04:17:01.799 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.799 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.800 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.800 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.800 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.800 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:17:01.800 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.801 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.801 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.801 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.801 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:17:01.801 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040273+0000", 2023-11-01T04:17:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3370000000000001e-06 2023-11-01T04:17:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040307+0000", 2023-11-01T04:17:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3714999999999998e-05 2023-11-01T04:17:01.804 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.804 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.804 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.804 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040331+0000", 2023-11-01T04:17:01.804 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4409e-05 2023-11-01T04:17:01.804 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.805 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.805 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.805 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140479+0000", 2023-11-01T04:17:01.805 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10014764900000001 2023-11-01T04:17:01.805 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.806 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.806 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.806 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.240734+0000", 2023-11-01T04:17:01.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100255645 2023-11-01T04:17:01.806 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.806 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.807 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.807 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.240751+0000", 2023-11-01T04:17:01.807 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6852000000000001e-05 2023-11-01T04:17:01.807 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.807 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.807 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.808 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341894+0000", 2023-11-01T04:17:01.808 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101143302 2023-11-01T04:17:01.808 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.808 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.808 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.809 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341949+0000", 2023-11-01T04:17:01.809 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4364999999999999e-05 2023-11-01T04:17:01.809 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.809 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.809 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.809 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.810 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.810 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 37/31 ECSubWrite(tid=23, reqid=client.4559.0:198, at_version=37'10, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:17:01.810 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.140534+0000", 2023-11-01T04:17:01.810 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 44.620147656999997, 2023-11-01T04:17:01.810 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202392451, 2023-11-01T04:17:01.811 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.811 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.811 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.811 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.811 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.811 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140534+0000", 2023-11-01T04:17:01.812 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.812 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.812 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.812 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.812 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140534+0000", 2023-11-01T04:17:01.813 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.813 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.813 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140536+0000", 2023-11-01T04:17:01.813 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.22e-06 2023-11-01T04:17:01.814 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.814 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.814 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.814 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140578+0000", 2023-11-01T04:17:01.814 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2715e-05 2023-11-01T04:17:01.815 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.815 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.815 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.815 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140603+0000", 2023-11-01T04:17:01.815 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4609000000000001e-05 2023-11-01T04:17:01.816 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.816 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.816 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.816 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140616+0000", 2023-11-01T04:17:01.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2763e-05 2023-11-01T04:17:01.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.817 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341493+0000", 2023-11-01T04:17:01.817 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.200877214 2023-11-01T04:17:01.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.818 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.818 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.818 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341508+0000", 2023-11-01T04:17:01.818 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5013e-05 2023-11-01T04:17:01.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.342869+0000", 2023-11-01T04:17:01.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001361183 2023-11-01T04:17:01.819 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.820 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.820 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.342927+0000", 2023-11-01T04:17:01.820 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7734e-05 2023-11-01T04:17:01.820 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.820 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.821 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.821 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.821 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.821 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 37/31 ECSubWrite(tid=13, reqid=client.4559.0:181, at_version=37'8, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:17:01.821 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.248764+0000", 2023-11-01T04:17:01.822 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 44.511918272000003, 2023-11-01T04:17:01.822 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20321630399999999, 2023-11-01T04:17:01.822 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.822 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.822 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.823 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.248764+0000", 2023-11-01T04:17:01.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.824 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.824 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.248764+0000", 2023-11-01T04:17:01.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.824 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.824 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.825 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.825 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.248766+0000", 2023-11-01T04:17:01.825 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8300000000000001e-06 2023-11-01T04:17:01.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.825 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.825 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.826 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.249255+0000", 2023-11-01T04:17:01.826 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00048892399999999996 2023-11-01T04:17:01.826 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.826 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.826 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.827 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.249286+0000", 2023-11-01T04:17:01.827 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1004999999999999e-05 2023-11-01T04:17:01.827 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.827 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.827 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.827 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.249302+0000", 2023-11-01T04:17:01.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6169999999999999e-05 2023-11-01T04:17:01.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.349516+0000", 2023-11-01T04:17:01.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10021408599999999 2023-11-01T04:17:01.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.829 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.829 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.349530+0000", 2023-11-01T04:17:01.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4472e-05 2023-11-01T04:17:01.830 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.830 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.830 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.830 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451931+0000", 2023-11-01T04:17:01.831 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10240034100000001 2023-11-01T04:17:01.831 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.831 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.831 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451980+0000", 2023-11-01T04:17:01.832 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9475999999999999e-05 2023-11-01T04:17:01.832 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.832 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.832 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.833 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 37/31 ECSubWrite(tid=14, reqid=client.4559.0:182, at_version=37'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:17:01.833 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.253221+0000", 2023-11-01T04:17:01.833 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 44.507461116000002, 2023-11-01T04:17:01.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19911255999999999, 2023-11-01T04:17:01.833 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.834 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.834 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253221+0000", 2023-11-01T04:17:01.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253221+0000", 2023-11-01T04:17:01.836 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.836 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.836 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.836 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.836 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253222+0000", 2023-11-01T04:17:01.836 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1570000000000001e-06 2023-11-01T04:17:01.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.837 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.837 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253418+0000", 2023-11-01T04:17:01.837 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019597500000000001 2023-11-01T04:17:01.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253443+0000", 2023-11-01T04:17:01.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5055999999999999e-05 2023-11-01T04:17:01.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.839 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.839 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.839 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.253459+0000", 2023-11-01T04:17:01.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5495999999999999e-05 2023-11-01T04:17:01.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.840 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451029+0000", 2023-11-01T04:17:01.840 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19757031799999999 2023-11-01T04:17:01.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.841 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.841 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451039+0000", 2023-11-01T04:17:01.841 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0062e-05 2023-11-01T04:17:01.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.841 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.842 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.842 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.452298+0000", 2023-11-01T04:17:01.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012584849999999999 2023-11-01T04:17:01.842 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.842 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.843 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.843 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.452334+0000", 2023-11-01T04:17:01.843 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6010999999999998e-05 2023-11-01T04:17:01.843 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.843 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.843 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.844 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.844 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 37/31 ECSubWrite(tid=15, reqid=client.4559.0:183, at_version=37'10, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:17:01.844 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.255859+0000", 2023-11-01T04:17:01.844 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 44.504823424000001, 2023-11-01T04:17:01.845 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19868693900000001, 2023-11-01T04:17:01.845 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.845 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.845 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.845 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.845 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.846 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255859+0000", 2023-11-01T04:17:01.846 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.846 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.846 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.847 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255859+0000", 2023-11-01T04:17:01.847 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.847 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.847 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.847 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.847 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255860+0000", 2023-11-01T04:17:01.848 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2300000000000001e-06 2023-11-01T04:17:01.848 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.848 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.848 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.848 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255894+0000", 2023-11-01T04:17:01.849 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3796e-05 2023-11-01T04:17:01.849 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.849 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.849 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.849 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255923+0000", 2023-11-01T04:17:01.849 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9352e-05 2023-11-01T04:17:01.850 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.850 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.850 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.850 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.255936+0000", 2023-11-01T04:17:01.850 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3232e-05 2023-11-01T04:17:01.851 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.851 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.851 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.851 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451650+0000", 2023-11-01T04:17:01.851 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19571368 2023-11-01T04:17:01.852 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.852 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.852 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.852 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.451665+0000", 2023-11-01T04:17:01.852 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5119e-05 2023-11-01T04:17:01.853 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.853 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.853 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.853 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.454503+0000", 2023-11-01T04:17:01.853 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028377400000000001 2023-11-01T04:17:01.854 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.854 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.854 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.854 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.454546+0000", 2023-11-01T04:17:01.854 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2790000000000002e-05 2023-11-01T04:17:01.854 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.855 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.855 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.855 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.855 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.855 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.9s1 58/31 ECSubWrite(tid=63, reqid=osd.7.0:63, at_version=58'57, trim_to=0'0, roll_forward_to=57'53))", 2023-11-01T04:17:01.856 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:38.447186+0000", 2023-11-01T04:17:01.856 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 23.313496202, 2023-11-01T04:17:01.856 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202892298, 2023-11-01T04:17:01.856 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.856 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.856 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.857 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.857 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.857 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447186+0000", 2023-11-01T04:17:01.857 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.857 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.857 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.858 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.858 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447186+0000", 2023-11-01T04:17:01.858 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.858 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.858 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.859 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.859 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447187+0000", 2023-11-01T04:17:01.859 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.333e-06 2023-11-01T04:17:01.859 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.859 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.859 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.860 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447243+0000", 2023-11-01T04:17:01.860 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5461999999999999e-05 2023-11-01T04:17:01.860 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.860 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.860 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.861 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447268+0000", 2023-11-01T04:17:01.861 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5286000000000002e-05 2023-11-01T04:17:01.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.861 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.861 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.861 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.447289+0000", 2023-11-01T04:17:01.862 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1060000000000002e-05 2023-11-01T04:17:01.862 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.862 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.862 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.862 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.647605+0000", 2023-11-01T04:17:01.862 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20031611899999999 2023-11-01T04:17:01.863 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.863 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.863 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.863 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.647619+0000", 2023-11-01T04:17:01.863 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4316e-05 2023-11-01T04:17:01.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.864 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.864 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.864 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.650003+0000", 2023-11-01T04:17:01.864 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002383994 2023-11-01T04:17:01.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.865 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.865 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.865 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.650078+0000", 2023-11-01T04:17:01.865 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4727999999999993e-05 2023-11-01T04:17:01.865 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.866 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.866 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.866 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.866 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.866 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.0s3 67/31 ECSubWrite(tid=35, reqid=osd.3.0:34, at_version=0'0, trim_to=0'0, roll_forward_to=67'19))", 2023-11-01T04:17:01.866 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.288148+0000", 2023-11-01T04:17:01.867 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 14.472534109, 2023-11-01T04:17:01.867 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20173064900000001, 2023-11-01T04:17:01.867 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.867 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.867 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.868 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.868 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.868 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.288148+0000", 2023-11-01T04:17:01.868 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.868 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.868 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.869 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.869 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.288148+0000", 2023-11-01T04:17:01.869 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.869 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.869 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.870 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.870 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.288150+0000", 2023-11-01T04:17:01.870 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4839999999999998e-06 2023-11-01T04:17:01.870 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.870 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.871 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.871 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.288174+0000", 2023-11-01T04:17:01.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3879999999999998e-05 2023-11-01T04:17:01.871 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.871 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.872 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.288206+0000", 2023-11-01T04:17:01.872 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1668999999999998e-05 2023-11-01T04:17:01.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.872 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.872 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.388354+0000", 2023-11-01T04:17:01.873 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10014759600000001 2023-11-01T04:17:01.873 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.873 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.873 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.873 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.488597+0000", 2023-11-01T04:17:01.874 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100242922 2023-11-01T04:17:01.874 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.874 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.874 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.874 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.488623+0000", 2023-11-01T04:17:01.874 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6713e-05 2023-11-01T04:17:01.875 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.875 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.875 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.875 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.489797+0000", 2023-11-01T04:17:01.875 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001174034 2023-11-01T04:17:01.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.876 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.876 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.489879+0000", 2023-11-01T04:17:01.876 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1351000000000001e-05 2023-11-01T04:17:01.876 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.877 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.877 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.877 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.877 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.877 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 67/31 ECSubWrite(tid=77, reqid=client.4559.0:537, at_version=67'58, trim_to=0'0, roll_forward_to=67'56))", 2023-11-01T04:17:01.878 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.705633+0000", 2023-11-01T04:17:01.878 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 14.055048821, 2023-11-01T04:17:01.878 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19831848199999999, 2023-11-01T04:17:01.878 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.878 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.879 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.879 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.879 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.879 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.705633+0000", 2023-11-01T04:17:01.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.880 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.705633+0000", 2023-11-01T04:17:01.880 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.881 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.881 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.881 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.705635+0000", 2023-11-01T04:17:01.881 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.793e-06 2023-11-01T04:17:01.881 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.882 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.882 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.882 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.706212+0000", 2023-11-01T04:17:01.882 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00057694199999999999 2023-11-01T04:17:01.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.883 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.883 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.706243+0000", 2023-11-01T04:17:01.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0593000000000003e-05 2023-11-01T04:17:01.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.884 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.884 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.884 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.706258+0000", 2023-11-01T04:17:01.884 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5556000000000001e-05 2023-11-01T04:17:01.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.884 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.885 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.885 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.802680+0000", 2023-11-01T04:17:01.885 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.096422161000000006 2023-11-01T04:17:01.885 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.885 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.886 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.886 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.802699+0000", 2023-11-01T04:17:01.886 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9023999999999999e-05 2023-11-01T04:17:01.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.886 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.887 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.887 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.903896+0000", 2023-11-01T04:17:01.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10119634499999999 2023-11-01T04:17:01.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.887 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.888 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.903952+0000", 2023-11-01T04:17:01.888 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6067999999999999e-05 2023-11-01T04:17:01.888 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.888 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.888 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.889 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpRead(3.6s2 72/68 ECSubRead(tid=104, to_read={3:6b2f2983:::smithi19039307-44:head=0,4096,0}, subchunks={3:6b2f2983:::smithi19039307-44:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:17:01.889 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:53.333737+0000", 2023-11-01T04:17:01.889 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 8.4269449139999999, 2023-11-01T04:17:01.890 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.200693012, 2023-11-01T04:17:01.890 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.890 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.890 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.891 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.891 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:53.333737+0000", 2023-11-01T04:17:01.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.891 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:53.333737+0000", 2023-11-01T04:17:01.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:53.333739+0000", 2023-11-01T04:17:01.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6869999999999999e-06 2023-11-01T04:17:01.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:53.333759+0000", 2023-11-01T04:17:01.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9675000000000001e-05 2023-11-01T04:17:01.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:53.333799+0000", 2023-11-01T04:17:01.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0794999999999998e-05 2023-11-01T04:17:01.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.895 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:53.433884+0000", 2023-11-01T04:17:01.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100084193 2023-11-01T04:17:01.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:53.534025+0000", 2023-11-01T04:17:01.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100141458 2023-11-01T04:17:01.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:53.534430+0000", 2023-11-01T04:17:01.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040520399999999999 2023-11-01T04:17:01.897 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.898 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.898 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.898 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 74/73 ECSubWrite(tid=111, reqid=osd.11.0:111, at_version=74'96, trim_to=0'0, roll_forward_to=0'0))", 2023-11-01T04:17:01.899 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:55.097767+0000", 2023-11-01T04:17:01.899 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 6.6629152559999998, 2023-11-01T04:17:01.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202752038, 2023-11-01T04:17:01.899 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.899 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.899 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.097767+0000", 2023-11-01T04:17:01.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.097767+0000", 2023-11-01T04:17:01.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.097767+0000", 2023-11-01T04:17:01.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.03e-07 2023-11-01T04:17:01.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.097813+0000", 2023-11-01T04:17:01.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5368000000000003e-05 2023-11-01T04:17:01.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.097829+0000", 2023-11-01T04:17:01.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6333000000000001e-05 2023-11-01T04:17:01.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.097846+0000", 2023-11-01T04:17:01.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7402e-05 2023-11-01T04:17:01.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.298672+0000", 2023-11-01T04:17:01.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20082581499999999 2023-11-01T04:17:01.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.298689+0000", 2023-11-01T04:17:01.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7068999999999999e-05 2023-11-01T04:17:01.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.300450+0000", 2023-11-01T04:17:01.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001761114 2023-11-01T04:17:01.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.300519+0000", 2023-11-01T04:17:01.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8534000000000005e-05 2023-11-01T04:17:01.909 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.909 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.909 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.910 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 74/73 ECSubWrite(tid=112, reqid=osd.11.0:112, at_version=74'98, trim_to=0'0, roll_forward_to=0'0))", 2023-11-01T04:17:01.910 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:55.097868+0000", 2023-11-01T04:17:01.910 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 6.6628143870000001, 2023-11-01T04:17:01.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20362171700000001, 2023-11-01T04:17:01.911 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.911 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.911 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.097868+0000", 2023-11-01T04:17:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.097868+0000", 2023-11-01T04:17:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.097868+0000", 2023-11-01T04:17:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2599999999999998e-07 2023-11-01T04:17:01.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.097900+0000", 2023-11-01T04:17:01.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2138999999999998e-05 2023-11-01T04:17:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.097915+0000", 2023-11-01T04:17:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5126000000000001e-05 2023-11-01T04:17:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.097930+0000", 2023-11-01T04:17:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4273e-05 2023-11-01T04:17:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.299587+0000", 2023-11-01T04:17:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.201657748 2023-11-01T04:17:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.299607+0000", 2023-11-01T04:17:01.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0069e-05 2023-11-01T04:17:01.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.301415+0000", 2023-11-01T04:17:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001807056 2023-11-01T04:17:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.301489+0000", 2023-11-01T04:17:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4880000000000001e-05 2023-11-01T04:17:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.bs2 74/31 ECSubWrite(tid=109, reqid=client.4559.0:626, at_version=0'0, trim_to=0'0, roll_forward_to=74'34))", 2023-11-01T04:17:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:55.302866+0000", 2023-11-01T04:17:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 6.4578161710000002, 2023-11-01T04:17:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20239728400000001, 2023-11-01T04:17:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.302866+0000", 2023-11-01T04:17:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.302866+0000", 2023-11-01T04:17:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.302868+0000", 2023-11-01T04:17:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.66e-06 2023-11-01T04:17:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.302889+0000", 2023-11-01T04:17:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0922000000000001e-05 2023-11-01T04:17:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.302915+0000", 2023-11-01T04:17:01.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6588999999999999e-05 2023-11-01T04:17:01.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.302943+0000", 2023-11-01T04:17:01.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8132000000000001e-05 2023-11-01T04:17:01.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.503352+0000", 2023-11-01T04:17:01.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20040834399999999 2023-11-01T04:17:01.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.503367+0000", 2023-11-01T04:17:01.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5369000000000001e-05 2023-11-01T04:17:01.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.505180+0000", 2023-11-01T04:17:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001813498 2023-11-01T04:17:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.505263+0000", 2023-11-01T04:17:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2769999999999995e-05 2023-11-01T04:17:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.bs2 77/31 ECSubWrite(tid=116, reqid=osd.10.0:115, at_version=0'0, trim_to=0'0, roll_forward_to=77'40))", 2023-11-01T04:17:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:58.527511+0000", 2023-11-01T04:17:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.2331708670000001, 2023-11-01T04:17:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.201865405, 2023-11-01T04:17:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.527511+0000", 2023-11-01T04:17:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.527511+0000", 2023-11-01T04:17:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.527513+0000", 2023-11-01T04:17:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1299999999999999e-06 2023-11-01T04:17:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.527540+0000", 2023-11-01T04:17:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7002999999999999e-05 2023-11-01T04:17:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.527581+0000", 2023-11-01T04:17:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0293999999999997e-05 2023-11-01T04:17:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.627755+0000", 2023-11-01T04:17:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100174414 2023-11-01T04:17:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.627903+0000", 2023-11-01T04:17:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014819400000000001 2023-11-01T04:17:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.627935+0000", 2023-11-01T04:17:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1764999999999998e-05 2023-11-01T04:17:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.729281+0000", 2023-11-01T04:17:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101345929 2023-11-01T04:17:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.729377+0000", 2023-11-01T04:17:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5675999999999997e-05 2023-11-01T04:17:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:647 3.9s1 3:9c07f031:::smithi19039307-43:head [create,setxattr ]PaibOQQZPXlPQ_TVRVsq`]`lQnrU_adXdhR`ia_ (33) in=40b,setxattr mga[\\pjjlr_f^pjpoNm[SiQinb`p]Uljsr\\PZamg (1614) in=40b,setxattr \\\\nNdqV]P_aoSQ]T][SZi`\\`emqRalsThkWkp]_O (1998) in=40b,setxattr QPRVcO^PYksRnjf^RqtasbqiemoSYpeimc]_kQOo (1505) in=40b,setxattr i`dWfsdYVZUqQmWleQeniob`plYiVWUWSsnsd`\\a (1017) in=40b,setxattr Qn`WN^VrYNfSm_nWkOlO[bZ_[Zci__kgsaZeg[_R (1614) in=40b,setxattr \\]NdaY]dOTpk\\bgtmgifQtgWtaqfd]r`TbktPRjh (19016) in=40b,setxattr NlQZaoabhi (349) in=10b,setxattr QbZ^bqgnd]QnP\\ZjeRjaP[UfZgdm[V^sRaRpnZYY (1963) in=40b,setxattr `]YWO_jgNfUs_SQ`RaqUhXmNY[TtZZbaUj\\NPtY^ (1190) in=40b,setxattr dnfqsNhbe]^ml]ekS[ST`fnk_kpiTUa]Y]ges`rW (17824) in=40b,setxattr RgOUnWhYeYcgnikgiQQWNr_OWg_k`YSaN^XQOdPc (1437) in=40b,setxattr QUmNqZgRZOU]WaWWgl\\pnRhtilqS`cQaScmgUp_e (1438) in=40b,setxattr TfW^Smr_geRfti]efnPdr`iWQUikeXOp`StYqtQd (12280) in=40b,setxattr ZtZ (349) in=3b,setxattr NQYfjd\\ (349) in=7b,setxattr aNaTblXeObdSN[ahZn`j[mcbUaSjeXOnc[dZ^ZZo (910) in=40b,setxattr deb]iS_OUlbmWR_sP^er[TdaV`cSqnnthSotq]YZ (7928) in=40b,setxattr ^s_gVdo]kXUSXb]aqYjbUYbRmcoYksXSWQdVsh (349) in=38b,setxattr h^YTbW\\dli\\^dffQaXRNaZQVkZPONqhhabmhR_[s (1822) in=40b] snapc 28=[28,27,26,22,1b,19,18,12] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e78)", 2023-11-01T04:17:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:58.841255+0000", 2023-11-01T04:17:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.9194271980000002, 2023-11-01T04:17:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.24042176, 2023-11-01T04:17:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 647 2023-11-01T04:17:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.841255+0000", 2023-11-01T04:17:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.841255+0000", 2023-11-01T04:17:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.841258+0000", 2023-11-01T04:17:01.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7099999999999999e-06 2023-11-01T04:17:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.841499+0000", 2023-11-01T04:17:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000241017 2023-11-01T04:17:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.841527+0000", 2023-11-01T04:17:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8622000000000001e-05 2023-11-01T04:17:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.841555+0000", 2023-11-01T04:17:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7798e-05 2023-11-01T04:17:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.841732+0000", 2023-11-01T04:17:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017735900000000001 2023-11-01T04:17:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op must wait for map", 2023-11-01T04:17:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.841758+0000", 2023-11-01T04:17:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5165999999999999e-05 2023-11-01T04:17:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.948792+0000", 2023-11-01T04:17:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.107034474 2023-11-01T04:17:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2023-11-01T04:17:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.948798+0000", 2023-11-01T04:17:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3860000000000004e-06 2023-11-01T04:17:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:00.081195+0000", 2023-11-01T04:17:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.132396741 2023-11-01T04:17:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:00.081677+0000", 2023-11-01T04:17:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00048148700000000001 2023-11-01T04:17:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 78/31 ECSubWrite(tid=118, reqid=osd.10.0:118, at_version=78'50, trim_to=0'0, roll_forward_to=76'48))", 2023-11-01T04:17:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:58.990333+0000", 2023-11-01T04:17:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.7703493319999999, 2023-11-01T04:17:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202305912, 2023-11-01T04:17:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.990333+0000", 2023-11-01T04:17:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.990333+0000", 2023-11-01T04:17:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.990334+0000", 2023-11-01T04:17:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5540000000000001e-06 2023-11-01T04:17:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.990382+0000", 2023-11-01T04:17:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7701000000000003e-05 2023-11-01T04:17:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.990496+0000", 2023-11-01T04:17:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011352499999999999 2023-11-01T04:17:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.990514+0000", 2023-11-01T04:17:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8756000000000001e-05 2023-11-01T04:17:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:59.190849+0000", 2023-11-01T04:17:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20033462199999999 2023-11-01T04:17:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:59.190865+0000", 2023-11-01T04:17:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6163000000000001e-05 2023-11-01T04:17:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:59.192567+0000", 2023-11-01T04:17:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001702364 2023-11-01T04:17:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:59.192639+0000", 2023-11-01T04:17:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1227000000000001e-05 2023-11-01T04:17:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es2 78/73 ECSubWrite(tid=130, reqid=osd.11.0:129, at_version=0'0, trim_to=0'0, roll_forward_to=78'120))", 2023-11-01T04:17:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:59.395666+0000", 2023-11-01T04:17:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.36501568, 2023-11-01T04:17:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20199007399999999, 2023-11-01T04:17:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:59.395666+0000", 2023-11-01T04:17:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:59.395666+0000", 2023-11-01T04:17:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:59.395667+0000", 2023-11-01T04:17:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0999999999999998e-07 2023-11-01T04:17:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:59.395685+0000", 2023-11-01T04:17:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8209000000000001e-05 2023-11-01T04:17:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:59.395700+0000", 2023-11-01T04:17:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4769e-05 2023-11-01T04:17:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:59.395714+0000", 2023-11-01T04:17:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.416e-05 2023-11-01T04:17:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:59.596040+0000", 2023-11-01T04:17:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20032525600000001 2023-11-01T04:17:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:59.596076+0000", 2023-11-01T04:17:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6643999999999997e-05 2023-11-01T04:17:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:59.597569+0000", 2023-11-01T04:17:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001492796 2023-11-01T04:17:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:59.597656+0000", 2023-11-01T04:17:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7529999999999997e-05 2023-11-01T04:17:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpRead(3.1s3 79/78 ECSubRead(tid=108, to_read={3:8900b4ea:::smithi19039307-31:head=0,700416,0}, subchunks={3:8900b4ea:::smithi19039307-31:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:17:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:00.284448+0000", 2023-11-01T04:17:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.476233868, 2023-11-01T04:17:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20070079800000001, 2023-11-01T04:17:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:00.284448+0000", 2023-11-01T04:17:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:00.284448+0000", 2023-11-01T04:17:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:00.284450+0000", 2023-11-01T04:17:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.37e-06 2023-11-01T04:17:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:00.284464+0000", 2023-11-01T04:17:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4779e-05 2023-11-01T04:17:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:00.284491+0000", 2023-11-01T04:17:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6842000000000002e-05 2023-11-01T04:17:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:00.384633+0000", 2023-11-01T04:17:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100142096 2023-11-01T04:17:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:00.484897+0000", 2023-11-01T04:17:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100263484 2023-11-01T04:17:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:00.485149+0000", 2023-11-01T04:17:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000252227 2023-11-01T04:17:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:01.990 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:01.991 INFO:tasks.rados.rados.0.smithi190.stdout:280: done (5 left) 2023-11-01T04:17:01.991 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 16 current snap is 24 2023-11-01T04:17:01.992 INFO:tasks.rados.rados.0.smithi190.stdout:275: expect (ObjNum 71 snap 8 seq_num 71) 2023-11-01T04:17:01.992 INFO:tasks.rados.rados.0.smithi190.stdout:277: finishing write tid 1 to smithi19039307-8 2023-11-01T04:17:01.992 INFO:tasks.rados.rados.0.smithi190.stdout:277: finishing write tid 2 to smithi19039307-8 2023-11-01T04:17:01.992 INFO:tasks.rados.rados.0.smithi190.stdout:277: finishing write tid 3 to smithi19039307-8 2023-11-01T04:17:01.992 INFO:tasks.rados.rados.0.smithi190.stdout:264: finishing write tid 6 to smithi19039307-46 2023-11-01T04:17:01.993 INFO:tasks.rados.rados.0.smithi190.stdout:264: oid 46 updating version 0 to 31 2023-11-01T04:17:01.993 INFO:tasks.rados.rados.0.smithi190.stdout:264: oid 46 version 31 is already newer than 29 2023-11-01T04:17:01.993 INFO:tasks.rados.rados.0.smithi190.stdout:264: oid 46 version 31 is already newer than 28 2023-11-01T04:17:01.993 INFO:tasks.rados.rados.0.smithi190.stdout:264: oid 46 version 31 is already newer than 30 2023-11-01T04:17:01.993 INFO:tasks.rados.rados.0.smithi190.stdout:264: oid 46 version 31 is already newer than 27 2023-11-01T04:17:01.994 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 46 v 31 (ObjNum 106 snap 24 seq_num 106) dirty exists 2023-11-01T04:17:01.994 INFO:tasks.rados.rados.0.smithi190.stdout:264: left oid 46 (ObjNum 106 snap 24 seq_num 106) 2023-11-01T04:17:01.994 INFO:tasks.rados.rados.0.smithi190.stdout:273: expect (ObjNum 86 snap 19 seq_num 86) 2023-11-01T04:17:01.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-01T04:17:01.995 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:01.988+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:02.079 INFO:tasks.rados.rados.0.smithi190.stdout:277: finishing write tid 4 to smithi19039307-8 2023-11-01T04:17:02.080 INFO:tasks.rados.rados.0.smithi190.stdout:277: finishing write tid 5 to smithi19039307-8 2023-11-01T04:17:02.080 INFO:tasks.rados.rados.0.smithi190.stdout:272: finishing copy_from racing read to smithi19039307-6 2023-11-01T04:17:02.080 INFO:tasks.rados.rados.0.smithi190.stdout:272: finishing copy_from to smithi19039307-6 2023-11-01T04:17:02.080 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 6 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2023-11-01T04:17:02.080 INFO:tasks.rados.rados.0.smithi190.stdout:277: finishing write tid 6 to smithi19039307-8 2023-11-01T04:17:02.080 INFO:tasks.rados.rados.0.smithi190.stdout:277: oid 8 updating version 0 to 91 2023-11-01T04:17:02.081 INFO:tasks.rados.rados.0.smithi190.stdout:277: oid 8 updating version 91 to 92 2023-11-01T04:17:02.081 INFO:tasks.rados.rados.0.smithi190.stdout:277: oid 8 version 92 is already newer than 90 2023-11-01T04:17:02.081 INFO:tasks.rados.rados.0.smithi190.stdout:277: oid 8 updating version 92 to 94 2023-11-01T04:17:02.081 INFO:tasks.rados.rados.0.smithi190.stdout:277: oid 8 version 94 is already newer than 93 2023-11-01T04:17:02.081 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 8 v 94 (ObjNum 108 snap 24 seq_num 108) dirty exists 2023-11-01T04:17:02.082 INFO:tasks.rados.rados.0.smithi190.stdout:277: left oid 8 (ObjNum 108 snap 24 seq_num 108) 2023-11-01T04:17:02.082 INFO:tasks.rados.rados.0.smithi190.stdout:281: seq_num 109 ranges {4472832=671744,5144576=753664,5898240=409600} 2023-11-01T04:17:02.120 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:17:02.093+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:17:02.153 INFO:tasks.rados.rados.0.smithi190.stdout:281: writing smithi19039307-16 from 4472832 to 5144576 tid 1 2023-11-01T04:17:02.159 INFO:tasks.rados.rados.0.smithi190.stdout:281: writing smithi19039307-16 from 5144576 to 5898240 tid 2 2023-11-01T04:17:02.163 INFO:tasks.rados.rados.0.smithi190.stdout:281: writing smithi19039307-16 from 5898240 to 6307840 tid 3 2023-11-01T04:17:02.163 INFO:tasks.rados.rados.0.smithi190.stdout:264: done (5 left) 2023-11-01T04:17:02.164 INFO:tasks.rados.rados.0.smithi190.stdout:272: done (4 left) 2023-11-01T04:17:02.164 INFO:tasks.rados.rados.0.smithi190.stdout:273: done (3 left) 2023-11-01T04:17:02.164 INFO:tasks.rados.rados.0.smithi190.stdout:275: done (2 left) 2023-11-01T04:17:02.164 INFO:tasks.rados.rados.0.smithi190.stdout:277: done (1 left) 2023-11-01T04:17:02.164 INFO:tasks.rados.rados.0.smithi190.stdout:282: read oid 3 snap -1 2023-11-01T04:17:02.164 INFO:tasks.rados.rados.0.smithi190.stdout:282: expect (ObjNum 6 snap 0 seq_num 6) 2023-11-01T04:17:02.165 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 20 current snap is 24 2023-11-01T04:17:02.165 INFO:tasks.rados.rados.0.smithi190.stdout:283: seq_num 110 ranges {1589248=688128,2277376=507904,2785280=655360} 2023-11-01T04:17:02.168 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:02.168 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:17:02.168 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2023-11-01T04:17:02.168 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:02.180 INFO:tasks.rados.rados.0.smithi190.stdout:283: writing smithi19039307-20 from 1589248 to 2277376 tid 1 2023-11-01T04:17:02.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-01T04:17:02.183 INFO:tasks.rados.rados.0.smithi190.stdout:283: writing smithi19039307-20 from 2277376 to 2785280 tid 2 2023-11-01T04:17:02.187 INFO:tasks.rados.rados.0.smithi190.stdout:283: writing smithi19039307-20 from 2785280 to 3440640 tid 3 2023-11-01T04:17:02.187 INFO:tasks.rados.rados.0.smithi190.stdout:284: snap_create 2023-11-01T04:17:02.193 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:17:02.191+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:17:02.293 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:02.310 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:02.310 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:17:02.311 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:17:02.311 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:17:02.311 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:02.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-01T04:17:02.395 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:17:02.392+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:17:02.449 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:02.450 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:17:02.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:17:02.450 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:17:02.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.451 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:166 3.8s0 3:1b945fb3:::smithi19039307-3:head [read 0~49152,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:17:02.451 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.317340+0000", 2023-11-01T04:17:02.451 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.130203729000002, 2023-11-01T04:17:02.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.405887145, 2023-11-01T04:17:02.451 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.452 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:02.452 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:02.452 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:02.452 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:02.453 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 166 2023-11-01T04:17:02.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.453 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.453 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317340+0000", 2023-11-01T04:17:02.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317340+0000", 2023-11-01T04:17:02.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317341+0000", 2023-11-01T04:17:02.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.077e-06 2023-11-01T04:17:02.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317350+0000", 2023-11-01T04:17:02.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0329999999999997e-06 2023-11-01T04:17:02.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317367+0000", 2023-11-01T04:17:02.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6612999999999998e-05 2023-11-01T04:17:02.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.458 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.458 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317380+0000", 2023-11-01T04:17:02.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2832e-05 2023-11-01T04:17:02.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.421004+0000", 2023-11-01T04:17:02.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.103624181 2023-11-01T04:17:02.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:02.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.421215+0000", 2023-11-01T04:17:02.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00021057199999999999 2023-11-01T04:17:02.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.723227+0000", 2023-11-01T04:17:02.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30201283699999998 2023-11-01T04:17:02.461 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.462 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.462 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.462 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:326 3.8s0 3:1514c4b8:::smithi19039307-15:head [append 0~376832] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:17:02.463 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.056621+0000", 2023-11-01T04:17:02.463 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 36.390922826000001, 2023-11-01T04:17:02.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43382757900000002, 2023-11-01T04:17:02.463 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.463 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:02.464 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:02.464 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:02.464 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:02.464 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 326 2023-11-01T04:17:02.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.465 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.056621+0000", 2023-11-01T04:17:02.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.056621+0000", 2023-11-01T04:17:02.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.056623+0000", 2023-11-01T04:17:02.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2640000000000001e-06 2023-11-01T04:17:02.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057077+0000", 2023-11-01T04:17:02.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00045417599999999998 2023-11-01T04:17:02.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057092+0000", 2023-11-01T04:17:02.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5138e-05 2023-11-01T04:17:02.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057103+0000", 2023-11-01T04:17:02.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1117999999999999e-05 2023-11-01T04:17:02.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.159853+0000", 2023-11-01T04:17:02.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10274968399999999 2023-11-01T04:17:02.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:02.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.159948+0000", 2023-11-01T04:17:02.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5097000000000001e-05 2023-11-01T04:17:02.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:02.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.160626+0000", 2023-11-01T04:17:02.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00067830199999999998 2023-11-01T04:17:02.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:02.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162313+0000", 2023-11-01T04:17:02.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016867799999999999 2023-11-01T04:17:02.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:02.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.490341+0000", 2023-11-01T04:17:02.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32802831100000002 2023-11-01T04:17:02.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.490449+0000", 2023-11-01T04:17:02.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010770899999999999 2023-11-01T04:17:02.476 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.476 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.476 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.477 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:327 3.8s0 3:1514c4b8:::smithi19039307-15:head [setxattr _header (56) in=63b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:17:02.477 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.057121+0000", 2023-11-01T04:17:02.477 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 36.390422766, 2023-11-01T04:17:02.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.63439970300000004, 2023-11-01T04:17:02.477 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.478 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:02.478 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:02.478 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:02.478 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:02.478 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 327 2023-11-01T04:17:02.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.479 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057121+0000", 2023-11-01T04:17:02.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057121+0000", 2023-11-01T04:17:02.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057122+0000", 2023-11-01T04:17:02.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8199999999999999e-07 2023-11-01T04:17:02.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057126+0000", 2023-11-01T04:17:02.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0439999999999997e-06 2023-11-01T04:17:02.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057134+0000", 2023-11-01T04:17:02.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.456e-06 2023-11-01T04:17:02.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057141+0000", 2023-11-01T04:17:02.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.861e-06 2023-11-01T04:17:02.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.161271+0000", 2023-11-01T04:17:02.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104129201 2023-11-01T04:17:02.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:02.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.161389+0000", 2023-11-01T04:17:02.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011806 2023-11-01T04:17:02.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:02.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.161906+0000", 2023-11-01T04:17:02.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00051756700000000003 2023-11-01T04:17:02.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:02.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162616+0000", 2023-11-01T04:17:02.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00070943799999999995 2023-11-01T04:17:02.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:02.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691275+0000", 2023-11-01T04:17:02.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52865958499999999 2023-11-01T04:17:02.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691521+0000", 2023-11-01T04:17:02.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024580899999999999 2023-11-01T04:17:02.490 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.490 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.490 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.491 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:328 3.8s0 3:1514c4b8:::smithi19039307-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:17:02.491 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.057152+0000", 2023-11-01T04:17:02.491 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 36.390392218999999, 2023-11-01T04:17:02.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83879504599999999, 2023-11-01T04:17:02.491 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.492 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:02.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:02.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:02.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:02.492 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 328 2023-11-01T04:17:02.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.493 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057152+0000", 2023-11-01T04:17:02.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057152+0000", 2023-11-01T04:17:02.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057152+0000", 2023-11-01T04:17:02.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5599999999999995e-07 2023-11-01T04:17:02.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057155+0000", 2023-11-01T04:17:02.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5050000000000002e-06 2023-11-01T04:17:02.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057163+0000", 2023-11-01T04:17:02.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1629999999999996e-06 2023-11-01T04:17:02.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057169+0000", 2023-11-01T04:17:02.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8930000000000002e-06 2023-11-01T04:17:02.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162181+0000", 2023-11-01T04:17:02.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105011939 2023-11-01T04:17:02.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:02.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162296+0000", 2023-11-01T04:17:02.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115392 2023-11-01T04:17:02.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691675+0000", 2023-11-01T04:17:02.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52937891999999998 2023-11-01T04:17:02.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:02.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691837+0000", 2023-11-01T04:17:02.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016180099999999999 2023-11-01T04:17:02.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.895947+0000", 2023-11-01T04:17:02.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.204109877 2023-11-01T04:17:02.503 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.503 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.503 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.504 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:339 3.8s0 3:19d11dee:::smithi19039307-39:head [append 0~737280 in=737280b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:17:02.504 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.185406+0000", 2023-11-01T04:17:02.504 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 36.262138292000003, 2023-11-01T04:17:02.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70798112199999996, 2023-11-01T04:17:02.504 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.505 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:02.505 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:02.505 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:02.505 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:02.505 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 339 2023-11-01T04:17:02.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.506 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.185406+0000", 2023-11-01T04:17:02.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.185406+0000", 2023-11-01T04:17:02.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.185408+0000", 2023-11-01T04:17:02.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9530000000000002e-06 2023-11-01T04:17:02.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186621+0000", 2023-11-01T04:17:02.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012132270000000001 2023-11-01T04:17:02.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186644+0000", 2023-11-01T04:17:02.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2878e-05 2023-11-01T04:17:02.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186663+0000", 2023-11-01T04:17:02.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9650999999999999e-05 2023-11-01T04:17:02.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186754+0000", 2023-11-01T04:17:02.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0606999999999995e-05 2023-11-01T04:17:02.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:02.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186902+0000", 2023-11-01T04:17:02.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000147448 2023-11-01T04:17:02.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:02.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.188122+0000", 2023-11-01T04:17:02.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001220732 2023-11-01T04:17:02.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.514 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:02.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.191051+0000", 2023-11-01T04:17:02.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029283669999999999 2023-11-01T04:17:02.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:02.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893331+0000", 2023-11-01T04:17:02.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70228026200000004 2023-11-01T04:17:02.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893387+0000", 2023-11-01T04:17:02.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5996999999999999e-05 2023-11-01T04:17:02.517 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.517 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.517 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.518 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:340 3.8s0 3:19d11dee:::smithi19039307-39:head [append 0~376832 in=376832b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:17:02.518 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.186694+0000", 2023-11-01T04:17:02.518 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 36.260850443000002, 2023-11-01T04:17:02.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70680890500000004, 2023-11-01T04:17:02.518 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.519 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:02.519 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:02.519 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:02.519 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:02.519 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 340 2023-11-01T04:17:02.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.520 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186694+0000", 2023-11-01T04:17:02.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186694+0000", 2023-11-01T04:17:02.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186695+0000", 2023-11-01T04:17:02.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3400000000000001e-06 2023-11-01T04:17:02.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187348+0000", 2023-11-01T04:17:02.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00065316300000000001 2023-11-01T04:17:02.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187364+0000", 2023-11-01T04:17:02.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5979999999999999e-05 2023-11-01T04:17:02.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.525 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187372+0000", 2023-11-01T04:17:02.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6599999999999995e-06 2023-11-01T04:17:02.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.189420+0000", 2023-11-01T04:17:02.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020482959999999998 2023-11-01T04:17:02.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:02.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.189500+0000", 2023-11-01T04:17:02.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9621000000000002e-05 2023-11-01T04:17:02.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:02.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.190386+0000", 2023-11-01T04:17:02.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00088641100000000003 2023-11-01T04:17:02.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:02.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.191553+0000", 2023-11-01T04:17:02.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001166413 2023-11-01T04:17:02.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:02.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893466+0000", 2023-11-01T04:17:02.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.701913911 2023-11-01T04:17:02.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893503+0000", 2023-11-01T04:17:02.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6109999999999998e-05 2023-11-01T04:17:02.531 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.531 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.531 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.532 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:341 3.8s0 3:19d11dee:::smithi19039307-39:head [setxattr _header (56) in=63b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:17:02.532 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.187391+0000", 2023-11-01T04:17:02.532 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 36.260152853000001, 2023-11-01T04:17:02.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70629933600000006, 2023-11-01T04:17:02.533 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.533 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:02.533 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:02.534 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:02.534 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:02.534 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 341 2023-11-01T04:17:02.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.535 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187391+0000", 2023-11-01T04:17:02.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187391+0000", 2023-11-01T04:17:02.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187392+0000", 2023-11-01T04:17:02.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7100000000000011e-07 2023-11-01T04:17:02.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187398+0000", 2023-11-01T04:17:02.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3329999999999996e-06 2023-11-01T04:17:02.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187407+0000", 2023-11-01T04:17:02.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6269999999999995e-06 2023-11-01T04:17:02.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187419+0000", 2023-11-01T04:17:02.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2336e-05 2023-11-01T04:17:02.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.265764+0000", 2023-11-01T04:17:02.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.078345021000000001 2023-11-01T04:17:02.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:02.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.265926+0000", 2023-11-01T04:17:02.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016117900000000001 2023-11-01T04:17:02.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:02.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.266578+0000", 2023-11-01T04:17:02.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00065216799999999995 2023-11-01T04:17:02.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:02.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.267885+0000", 2023-11-01T04:17:02.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013072699999999999 2023-11-01T04:17:02.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:02.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893617+0000", 2023-11-01T04:17:02.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62573200299999998 2023-11-01T04:17:02.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893691+0000", 2023-11-01T04:17:02.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3428000000000002e-05 2023-11-01T04:17:02.547 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.547 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.547 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.548 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:342 3.8s0 3:19d11dee:::smithi19039307-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:17:02.548 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.187433+0000", 2023-11-01T04:17:02.548 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 36.260110926999999, 2023-11-01T04:17:02.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.009816936, 2023-11-01T04:17:02.549 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.549 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:02.549 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:02.549 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:02.549 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:02.550 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 342 2023-11-01T04:17:02.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.550 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187433+0000", 2023-11-01T04:17:02.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187433+0000", 2023-11-01T04:17:02.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187434+0000", 2023-11-01T04:17:02.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.75e-07 2023-11-01T04:17:02.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187437+0000", 2023-11-01T04:17:02.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1870000000000001e-06 2023-11-01T04:17:02.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187446+0000", 2023-11-01T04:17:02.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0499999999999997e-06 2023-11-01T04:17:02.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.287537+0000", 2023-11-01T04:17:02.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10009085 2023-11-01T04:17:02.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.288584+0000", 2023-11-01T04:17:02.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001047463 2023-11-01T04:17:02.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:02.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.288883+0000", 2023-11-01T04:17:02.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029857700000000001 2023-11-01T04:17:02.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.894227+0000", 2023-11-01T04:17:02.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60534433899999995 2023-11-01T04:17:02.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:02.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.894404+0000", 2023-11-01T04:17:02.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000176732 2023-11-01T04:17:02.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.197250+0000", 2023-11-01T04:17:02.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.302846163 2023-11-01T04:17:02.562 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.563 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.563 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.563 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=28, last_complete=46'20, committed=1, applied=1))", 2023-11-01T04:17:02.564 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.390586+0000", 2023-11-01T04:17:02.564 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 36.056958041000001, 2023-11-01T04:17:02.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30212400700000003, 2023-11-01T04:17:02.564 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.564 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:02.565 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390586+0000", 2023-11-01T04:17:02.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390586+0000", 2023-11-01T04:17:02.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390587+0000", 2023-11-01T04:17:02.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9999999999999997e-07 2023-11-01T04:17:02.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390592+0000", 2023-11-01T04:17:02.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9400000000000001e-06 2023-11-01T04:17:02.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390607+0000", 2023-11-01T04:17:02.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5216e-05 2023-11-01T04:17:02.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390617+0000", 2023-11-01T04:17:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8260000000000005e-06 2023-11-01T04:17:02.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.692681+0000", 2023-11-01T04:17:02.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30206392300000001 2023-11-01T04:17:02.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.692710+0000", 2023-11-01T04:17:02.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9502e-05 2023-11-01T04:17:02.572 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.573 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=29, last_complete=46'21, committed=1, applied=1))", 2023-11-01T04:17:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.391131+0000", 2023-11-01T04:17:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 36.056412905000002, 2023-11-01T04:17:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30166817800000001, 2023-11-01T04:17:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.574 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391131+0000", 2023-11-01T04:17:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391131+0000", 2023-11-01T04:17:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391132+0000", 2023-11-01T04:17:02.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5000000000000001e-07 2023-11-01T04:17:02.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391138+0000", 2023-11-01T04:17:02.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1999999999999999e-06 2023-11-01T04:17:02.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391156+0000", 2023-11-01T04:17:02.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7252999999999999e-05 2023-11-01T04:17:02.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391165+0000", 2023-11-01T04:17:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3530000000000001e-06 2023-11-01T04:17:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.692767+0000", 2023-11-01T04:17:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30160192600000002 2023-11-01T04:17:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.692799+0000", 2023-11-01T04:17:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2496000000000002e-05 2023-11-01T04:17:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.582 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:17:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:17:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 36.055205497000003, 2023-11-01T04:17:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40054957200000002, 2023-11-01T04:17:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:17:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:17:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:17:02.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7400000000000002e-07 2023-11-01T04:17:02.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392345+0000", 2023-11-01T04:17:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8690000000000001e-06 2023-11-01T04:17:02.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392362+0000", 2023-11-01T04:17:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6194000000000001e-05 2023-11-01T04:17:02.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392372+0000", 2023-11-01T04:17:02.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0692e-05 2023-11-01T04:17:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792857+0000", 2023-11-01T04:17:02.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40048482800000001 2023-11-01T04:17:02.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792888+0000", 2023-11-01T04:17:02.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1115000000000003e-05 2023-11-01T04:17:02.593 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.593 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.593 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.593 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=28, last_complete=46'20, committed=1, applied=1))", 2023-11-01T04:17:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.393704+0000", 2023-11-01T04:17:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 36.053839979999999, 2023-11-01T04:17:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39925663099999997, 2023-11-01T04:17:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.594 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393704+0000", 2023-11-01T04:17:02.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393704+0000", 2023-11-01T04:17:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393705+0000", 2023-11-01T04:17:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3699999999999999e-07 2023-11-01T04:17:02.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393711+0000", 2023-11-01T04:17:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6899999999999997e-06 2023-11-01T04:17:02.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393726+0000", 2023-11-01T04:17:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5559000000000001e-05 2023-11-01T04:17:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393735+0000", 2023-11-01T04:17:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2830000000000001e-06 2023-11-01T04:17:02.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792931+0000", 2023-11-01T04:17:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39919643599999999 2023-11-01T04:17:02.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792961+0000", 2023-11-01T04:17:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9726e-05 2023-11-01T04:17:02.602 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.603 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=29, last_complete=46'21, committed=1, applied=1))", 2023-11-01T04:17:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:17:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 36.053781102000002, 2023-11-01T04:17:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39925449800000001, 2023-11-01T04:17:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.604 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:17:02.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:17:02.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:17:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7700000000000005e-07 2023-11-01T04:17:02.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393770+0000", 2023-11-01T04:17:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.28e-06 2023-11-01T04:17:02.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393779+0000", 2023-11-01T04:17:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6760000000000002e-06 2023-11-01T04:17:02.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393786+0000", 2023-11-01T04:17:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8229999999999997e-06 2023-11-01T04:17:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792996+0000", 2023-11-01T04:17:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39921008299999999 2023-11-01T04:17:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.793017+0000", 2023-11-01T04:17:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1158999999999999e-05 2023-11-01T04:17:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.612 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:17:02.613 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.394729+0000", 2023-11-01T04:17:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 36.052815195999997, 2023-11-01T04:17:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.49843099299999999, 2023-11-01T04:17:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:02.614 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394729+0000", 2023-11-01T04:17:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394729+0000", 2023-11-01T04:17:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394730+0000", 2023-11-01T04:17:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.093e-06 2023-11-01T04:17:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394738+0000", 2023-11-01T04:17:02.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2300000000000008e-06 2023-11-01T04:17:02.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394760+0000", 2023-11-01T04:17:02.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1679999999999999e-05 2023-11-01T04:17:02.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.395298+0000", 2023-11-01T04:17:02.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053837199999999996 2023-11-01T04:17:02.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893130+0000", 2023-11-01T04:17:02.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.49783206600000002 2023-11-01T04:17:02.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893160+0000", 2023-11-01T04:17:02.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9552000000000001e-05 2023-11-01T04:17:02.622 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.622 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.622 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.623 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:17:02.623 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.468843+0000", 2023-11-01T04:17:02.623 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.978701346000001, 2023-11-01T04:17:02.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42441382100000002, 2023-11-01T04:17:02.624 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.624 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:02.624 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468843+0000", 2023-11-01T04:17:02.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468843+0000", 2023-11-01T04:17:02.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468845+0000", 2023-11-01T04:17:02.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.21e-06 2023-11-01T04:17:02.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468857+0000", 2023-11-01T04:17:02.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1678999999999999e-05 2023-11-01T04:17:02.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468888+0000", 2023-11-01T04:17:02.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1788999999999997e-05 2023-11-01T04:17:02.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569025+0000", 2023-11-01T04:17:02.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100136424 2023-11-01T04:17:02.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893225+0000", 2023-11-01T04:17:02.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32420036899999999 2023-11-01T04:17:02.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893257+0000", 2023-11-01T04:17:02.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1350000000000003e-05 2023-11-01T04:17:02.632 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.632 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.632 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.633 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=28, last_complete=46'20, committed=1, applied=1))", 2023-11-01T04:17:02.633 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.569161+0000", 2023-11-01T04:17:02.633 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.878383067999998, 2023-11-01T04:17:02.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32424099299999998, 2023-11-01T04:17:02.634 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.634 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:02.634 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569161+0000", 2023-11-01T04:17:02.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569161+0000", 2023-11-01T04:17:02.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569162+0000", 2023-11-01T04:17:02.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3400000000000001e-06 2023-11-01T04:17:02.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569173+0000", 2023-11-01T04:17:02.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0293e-05 2023-11-01T04:17:02.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569197+0000", 2023-11-01T04:17:02.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4156e-05 2023-11-01T04:17:02.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569213+0000", 2023-11-01T04:17:02.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6455000000000001e-05 2023-11-01T04:17:02.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893271+0000", 2023-11-01T04:17:02.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32405778099999999 2023-11-01T04:17:02.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893402+0000", 2023-11-01T04:17:02.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000130968 2023-11-01T04:17:02.642 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.642 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.642 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.643 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=29, last_complete=46'21, committed=1, applied=1))", 2023-11-01T04:17:02.643 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.569263+0000", 2023-11-01T04:17:02.643 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.878280588000003, 2023-11-01T04:17:02.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32424595, 2023-11-01T04:17:02.644 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.644 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:02.644 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569263+0000", 2023-11-01T04:17:02.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569263+0000", 2023-11-01T04:17:02.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569264+0000", 2023-11-01T04:17:02.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7700000000000004e-07 2023-11-01T04:17:02.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569269+0000", 2023-11-01T04:17:02.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9490000000000004e-06 2023-11-01T04:17:02.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569282+0000", 2023-11-01T04:17:02.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3169999999999999e-05 2023-11-01T04:17:02.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569293+0000", 2023-11-01T04:17:02.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0526000000000001e-05 2023-11-01T04:17:02.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893418+0000", 2023-11-01T04:17:02.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32412480999999999 2023-11-01T04:17:02.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893509+0000", 2023-11-01T04:17:02.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1817999999999994e-05 2023-11-01T04:17:02.652 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.652 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.652 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.653 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:17:02.653 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.569435+0000", 2023-11-01T04:17:02.653 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.878108613999999, 2023-11-01T04:17:02.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32468964500000003, 2023-11-01T04:17:02.654 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.654 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:02.654 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569435+0000", 2023-11-01T04:17:02.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569435+0000", 2023-11-01T04:17:02.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569436+0000", 2023-11-01T04:17:02.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6700000000000003e-07 2023-11-01T04:17:02.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569441+0000", 2023-11-01T04:17:02.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3059999999999997e-06 2023-11-01T04:17:02.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569461+0000", 2023-11-01T04:17:02.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9856000000000001e-05 2023-11-01T04:17:02.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.669600+0000", 2023-11-01T04:17:02.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100139114 2023-11-01T04:17:02.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893551+0000", 2023-11-01T04:17:02.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.223950963 2023-11-01T04:17:02.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.894125+0000", 2023-11-01T04:17:02.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00057373899999999998 2023-11-01T04:17:02.662 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.663 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.663 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.663 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(osd.12.22:5 3.8s0 3:19d11dee:::smithi19039307-39:head [assert-version v22,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:17:02.663 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.454207+0000", 2023-11-01T04:17:02.664 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 34.993336665000001, 2023-11-01T04:17:02.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35587170800000001, 2023-11-01T04:17:02.664 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.664 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:02.664 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454207+0000", 2023-11-01T04:17:02.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454207+0000", 2023-11-01T04:17:02.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454210+0000", 2023-11-01T04:17:02.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2299999999999998e-06 2023-11-01T04:17:02.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454219+0000", 2023-11-01T04:17:02.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0469999999999999e-06 2023-11-01T04:17:02.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454239+0000", 2023-11-01T04:17:02.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0499000000000001e-05 2023-11-01T04:17:02.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454256+0000", 2023-11-01T04:17:02.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6993000000000001e-05 2023-11-01T04:17:02.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454343+0000", 2023-11-01T04:17:02.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.687e-05 2023-11-01T04:17:02.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:02.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454435+0000", 2023-11-01T04:17:02.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2019000000000001e-05 2023-11-01T04:17:02.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.810079+0000", 2023-11-01T04:17:02.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35564404999999999 2023-11-01T04:17:02.673 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.673 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.673 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.674 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.0s4 68/31 ECSubWrite(tid=36, reqid=osd.3.0:36, at_version=68'21, trim_to=0'0, roll_forward_to=67'19))", 2023-11-01T04:17:02.674 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.894669+0000", 2023-11-01T04:17:02.674 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 14.552874585, 2023-11-01T04:17:02.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31005245300000001, 2023-11-01T04:17:02.675 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:02.675 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:02.675 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:02.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:02.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894669+0000", 2023-11-01T04:17:02.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:02.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894669+0000", 2023-11-01T04:17:02.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:02.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:02.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894670+0000", 2023-11-01T04:17:02.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9399999999999995e-07 2023-11-01T04:17:02.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:02.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894722+0000", 2023-11-01T04:17:02.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1962000000000002e-05 2023-11-01T04:17:02.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:02.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894772+0000", 2023-11-01T04:17:02.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9771999999999997e-05 2023-11-01T04:17:02.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:02.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.994910+0000", 2023-11-01T04:17:02.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100138189 2023-11-01T04:17:02.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:02.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.104098+0000", 2023-11-01T04:17:02.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10918811 2023-11-01T04:17:02.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:02.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.104110+0000", 2023-11-01T04:17:02.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1787000000000001e-05 2023-11-01T04:17:02.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:02.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.204645+0000", 2023-11-01T04:17:02.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100534978 2023-11-01T04:17:02.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:02.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:02.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:02.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.204722+0000", 2023-11-01T04:17:02.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7161000000000002e-05 2023-11-01T04:17:02.685 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.685 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.685 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.685 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:02.685 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:02.686 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:02.687 INFO:tasks.rados.rados.0.smithi190.stdout:282: expect (ObjNum 6 snap 0 seq_num 6) 2023-11-01T04:17:02.688 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-01T04:17:02.688 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:02.684+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:02.709 INFO:tasks.rados.rados.0.smithi190.stdout:281: finishing write tid 1 to smithi19039307-16 2023-11-01T04:17:02.788 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:17:02.789+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:17:02.861 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:02.862 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:02.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-01T04:17:02.880 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-11-01T04:17:02.888 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:17:02.889+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:17:02.892 INFO:tasks.rados.rados.0.smithi190.stdout:282: done (3 left) 2023-11-01T04:17:02.893 INFO:tasks.rados.rados.0.smithi190.stdout:284: done (2 left) 2023-11-01T04:17:02.893 INFO:tasks.rados.rados.0.smithi190.stdout:285: delete oid 30 current snap is 25 2023-11-01T04:17:02.949 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2023-11-01T04:17:02.965 INFO:tasks.rados.rados.0.smithi190.stdout:281: finishing write tid 2 to smithi19039307-16 2023-11-01T04:17:02.966 INFO:tasks.rados.rados.0.smithi190.stdout:281: finishing write tid 3 to smithi19039307-16 2023-11-01T04:17:02.966 INFO:tasks.rados.rados.0.smithi190.stdout:281: finishing write tid 4 to smithi19039307-16 2023-11-01T04:17:02.968 INFO:tasks.rados.rados.0.smithi190.stdout:283: finishing write tid 1 to smithi19039307-20 2023-11-01T04:17:02.968 INFO:tasks.rados.rados.0.smithi190.stdout:283: finishing write tid 2 to smithi19039307-20 2023-11-01T04:17:02.969 INFO:tasks.rados.rados.0.smithi190.stdout:283: finishing write tid 3 to smithi19039307-20 2023-11-01T04:17:02.969 INFO:tasks.rados.rados.0.smithi190.stdout:283: finishing write tid 4 to smithi19039307-20 2023-11-01T04:17:02.989 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:02.987+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:02.996 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:02.997 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:03.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-01T04:17:03.061 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:03.064 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:03.064 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['config', 'set', 'filestore_inject_stall', '3'] 2023-11-01T04:17:03.070 INFO:tasks.rados.rados.0.smithi190.stdout:281: finishing write tid 5 to smithi19039307-16 2023-11-01T04:17:03.070 INFO:tasks.rados.rados.0.smithi190.stdout:281: oid 16 updating version 0 to 112 2023-11-01T04:17:03.070 INFO:tasks.rados.rados.0.smithi190.stdout:281: oid 16 version 112 is already newer than 109 2023-11-01T04:17:03.070 INFO:tasks.rados.rados.0.smithi190.stdout:281: oid 16 version 112 is already newer than 111 2023-11-01T04:17:03.070 INFO:tasks.rados.rados.0.smithi190.stdout:281: oid 16 version 112 is already newer than 110 2023-11-01T04:17:03.071 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 16 v 112 (ObjNum 109 snap 24 seq_num 109) dirty exists 2023-11-01T04:17:03.071 INFO:tasks.rados.rados.0.smithi190.stdout:281: left oid 16 (ObjNum 109 snap 24 seq_num 109) 2023-11-01T04:17:03.090 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:17:03.088+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:17:03.131 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:03.132 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:03.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-01T04:17:03.191 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:17:03.188+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:17:03.267 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:03.268 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:03.273 INFO:tasks.rados.rados.0.smithi190.stdout:283: finishing write tid 5 to smithi19039307-20 2023-11-01T04:17:03.273 INFO:tasks.rados.rados.0.smithi190.stdout:283: oid 20 updating version 0 to 115 2023-11-01T04:17:03.273 INFO:tasks.rados.rados.0.smithi190.stdout:283: oid 20 updating version 115 to 117 2023-11-01T04:17:03.273 INFO:tasks.rados.rados.0.smithi190.stdout:283: oid 20 version 117 is already newer than 114 2023-11-01T04:17:03.274 INFO:tasks.rados.rados.0.smithi190.stdout:283: oid 20 version 117 is already newer than 116 2023-11-01T04:17:03.274 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 20 v 117 (ObjNum 110 snap 24 seq_num 110) dirty exists 2023-11-01T04:17:03.274 INFO:tasks.rados.rados.0.smithi190.stdout:283: left oid 20 (ObjNum 110 snap 24 seq_num 110) 2023-11-01T04:17:03.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-01T04:17:03.291 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~8s 2023-11-01T04:17:03.291 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-11-01T04:17:03.292 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:03.291+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:03.317 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-11-01T04:17:03.341 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:17:03.341 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-01T04:17:03.375 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:17:03.376 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-01T04:17:03.393 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:03.391+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:03.398 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:17:03.399 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-01T04:17:03.408 INFO:tasks.rados.rados.0.smithi190.stdout:281: done (2 left) 2023-11-01T04:17:03.408 INFO:tasks.rados.rados.0.smithi190.stdout:283: done (1 left) 2023-11-01T04:17:03.408 INFO:tasks.rados.rados.0.smithi190.stdout:285: done (0 left) 2023-11-01T04:17:03.408 INFO:tasks.rados.rados.0.smithi190.stdout:286: read oid 13 snap -1 2023-11-01T04:17:03.409 INFO:tasks.rados.rados.0.smithi190.stdout:286: expect (ObjNum 98 snap 21 seq_num 98) 2023-11-01T04:17:03.409 INFO:tasks.rados.rados.0.smithi190.stdout:287: setattr oid 26 current snap is 25 2023-11-01T04:17:03.410 INFO:tasks.rados.rados.0.smithi190.stdout:288: copy_from oid 32 from oid 7 current snap is 25 2023-11-01T04:17:03.410 INFO:tasks.rados.rados.0.smithi190.stdout:289: read oid 22 snap -1 2023-11-01T04:17:03.410 INFO:tasks.rados.rados.0.smithi190.stdout:289: expect (ObjNum 21 snap 0 seq_num 21) 2023-11-01T04:17:03.411 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 15 current snap is 25 2023-11-01T04:17:03.411 INFO:tasks.rados.rados.0.smithi190.stdout:290: seq_num 112 ranges {81920=720896,802816=802816,1605632=360448} 2023-11-01T04:17:03.416 INFO:tasks.rados.rados.0.smithi190.stdout:290: writing smithi19039307-15 from 81920 to 802816 tid 1 2023-11-01T04:17:03.424 INFO:tasks.rados.rados.0.smithi190.stdout:290: writing smithi19039307-15 from 802816 to 1605632 tid 2 2023-11-01T04:17:03.428 INFO:tasks.rados.rados.0.smithi190.stdout:290: writing smithi19039307-15 from 1605632 to 1966080 tid 3 2023-11-01T04:17:03.428 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 49 current snap is 25 2023-11-01T04:17:03.428 INFO:tasks.rados.rados.0.smithi190.stdout:291: seq_num 113 ranges {1114112=737280,1851392=557056,2408448=606208} 2023-11-01T04:17:03.429 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:17:03.429 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-01T04:17:03.429 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.12 2023-11-01T04:17:03.443 INFO:tasks.rados.rados.0.smithi190.stdout:291: writing smithi19039307-49 from 1114112 to 1851392 tid 1 2023-11-01T04:17:03.447 INFO:tasks.rados.rados.0.smithi190.stdout:291: writing smithi19039307-49 from 1851392 to 2408448 tid 2 2023-11-01T04:17:03.452 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:03.452 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:03.453 INFO:tasks.rados.rados.0.smithi190.stdout:291: writing smithi19039307-49 from 2408448 to 3014656 tid 3 2023-11-01T04:17:03.453 INFO:tasks.rados.rados.0.smithi190.stdout:292: rollback oid 33 current snap is 25 2023-11-01T04:17:03.453 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 33 to 15 2023-11-01T04:17:03.453 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 30 current snap is 25 2023-11-01T04:17:03.453 INFO:tasks.rados.rados.0.smithi190.stdout:293: seq_num 114 ranges {0=753664,753664=688128,1441792=475136} 2023-11-01T04:17:03.458 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:17:03.459 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-01T04:17:03.459 INFO:tasks.rados.rados.0.smithi190.stdout:293: writing smithi19039307-30 from 0 to 753664 tid 1 2023-11-01T04:17:03.464 INFO:tasks.rados.rados.0.smithi190.stdout:293: writing smithi19039307-30 from 753664 to 1441792 tid 2 2023-11-01T04:17:03.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-01T04:17:03.468 INFO:tasks.rados.rados.0.smithi190.stdout:293: writing smithi19039307-30 from 1441792 to 1916928 tid 3 2023-11-01T04:17:03.468 INFO:tasks.rados.rados.0.smithi190.stdout:294: read oid 37 snap -1 2023-11-01T04:17:03.468 INFO:tasks.rados.rados.0.smithi190.stdout:294: expect (ObjNum 87 snap 20 seq_num 87) 2023-11-01T04:17:03.469 INFO:tasks.rados.rados.0.smithi190.stdout:295: rollback oid 8 current snap is 25 2023-11-01T04:17:03.469 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 8 to 24 2023-11-01T04:17:03.469 INFO:tasks.rados.rados.0.smithi190.stdout:296: rmattr oid 46 current snap is 25 2023-11-01T04:17:03.469 INFO:tasks.rados.rados.0.smithi190.stdout:296: done (10 left) 2023-11-01T04:17:03.469 INFO:tasks.rados.rados.0.smithi190.stdout:297: copy_from oid 3 from oid 5 current snap is 25 2023-11-01T04:17:03.470 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 46 current snap is 25 2023-11-01T04:17:03.470 INFO:tasks.rados.rados.0.smithi190.stdout:298: seq_num 115 ranges {2555904=770048,3325952=704512,4030464=458752} 2023-11-01T04:17:03.485 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:17:03.486 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-01T04:17:03.492 INFO:tasks.rados.rados.0.smithi190.stdout:298: writing smithi19039307-46 from 2555904 to 3325952 tid 1 2023-11-01T04:17:03.497 INFO:tasks.rados.rados.0.smithi190.stdout:298: writing smithi19039307-46 from 3325952 to 4030464 tid 2 2023-11-01T04:17:03.500 INFO:tasks.rados.rados.0.smithi190.stdout:298: writing smithi19039307-46 from 4030464 to 4489216 tid 3 2023-11-01T04:17:03.500 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 31 current snap is 25 2023-11-01T04:17:03.500 INFO:tasks.rados.rados.0.smithi190.stdout:299: seq_num 116 ranges {2801664=786432,3588096=458752,4046848=442368,4489216=262144} 2023-11-01T04:17:03.511 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:17:03.512 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-01T04:17:03.514 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:17:03.512+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:17:03.516 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-01T04:17:03.517 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-01T04:17:03.522 INFO:tasks.rados.rados.0.smithi190.stdout:299: writing smithi19039307-31 from 2801664 to 3588096 tid 1 2023-11-01T04:17:03.525 INFO:tasks.rados.rados.0.smithi190.stdout:299: writing smithi19039307-31 from 3588096 to 4046848 tid 2 2023-11-01T04:17:03.527 INFO:tasks.rados.rados.0.smithi190.stdout:299: writing smithi19039307-31 from 4046848 to 4489216 tid 3 2023-11-01T04:17:03.529 INFO:tasks.rados.rados.0.smithi190.stdout:299: writing smithi19039307-31 from 4489216 to 4751360 tid 4 2023-11-01T04:17:03.529 INFO:tasks.rados.rados.0.smithi190.stdout:300: read oid 12 snap 23 2023-11-01T04:17:03.529 INFO:tasks.rados.rados.0.smithi190.stdout:300: expect (ObjNum 11 snap 0 seq_num 11) 2023-11-01T04:17:03.530 INFO:tasks.rados.rados.0.smithi190.stdout:301: snap_create 2023-11-01T04:17:03.530 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 26 v 56 (ObjNum 25 snap 0 seq_num 25) dirty exists 2023-11-01T04:17:03.594 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:17:03.591+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:17:03.596 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:03.596 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:17:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:17:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:17:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:17:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138761+0000", 2023-11-01T04:17:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 58.455079066000003, 2023-11-01T04:17:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61681865999999996, 2023-11-01T04:17:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:17:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:17:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 114 2023-11-01T04:17:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138761+0000", 2023-11-01T04:17:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138761+0000", 2023-11-01T04:17:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138762+0000", 2023-11-01T04:17:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4300000000000003e-07 2023-11-01T04:17:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138768+0000", 2023-11-01T04:17:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7729999999999996e-06 2023-11-01T04:17:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138777+0000", 2023-11-01T04:17:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1929999999999999e-06 2023-11-01T04:17:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138784+0000", 2023-11-01T04:17:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3830000000000003e-06 2023-11-01T04:17:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.248094+0000", 2023-11-01T04:17:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10930951699999999 2023-11-01T04:17:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.248210+0000", 2023-11-01T04:17:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011579299999999999 2023-11-01T04:17:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:17:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.248407+0000", 2023-11-01T04:17:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019723000000000001 2023-11-01T04:17:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:17:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352490+0000", 2023-11-01T04:17:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104083393 2023-11-01T04:17:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:17:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755390+0000", 2023-11-01T04:17:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40290009500000001 2023-11-01T04:17:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755474+0000", 2023-11-01T04:17:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3323000000000002e-05 2023-11-01T04:17:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755580+0000", 2023-11-01T04:17:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010631700000000001 2023-11-01T04:17:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:17:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138795+0000", 2023-11-01T04:17:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 58.455044907999998, 2023-11-01T04:17:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61707305000000001, 2023-11-01T04:17:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:17:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:17:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 115 2023-11-01T04:17:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138795+0000", 2023-11-01T04:17:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138795+0000", 2023-11-01T04:17:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138796+0000", 2023-11-01T04:17:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3300000000000002e-07 2023-11-01T04:17:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138931+0000", 2023-11-01T04:17:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013525000000000001 2023-11-01T04:17:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138947+0000", 2023-11-01T04:17:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5478999999999999e-05 2023-11-01T04:17:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138956+0000", 2023-11-01T04:17:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1530000000000003e-06 2023-11-01T04:17:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.249005+0000", 2023-11-01T04:17:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.110048914 2023-11-01T04:17:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.249150+0000", 2023-11-01T04:17:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014491600000000001 2023-11-01T04:17:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:17:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.249351+0000", 2023-11-01T04:17:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000201516 2023-11-01T04:17:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:17:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352738+0000", 2023-11-01T04:17:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10338632 2023-11-01T04:17:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:17:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755668+0000", 2023-11-01T04:17:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40293035700000002 2023-11-01T04:17:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755765+0000", 2023-11-01T04:17:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7081000000000001e-05 2023-11-01T04:17:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755868+0000", 2023-11-01T04:17:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010343100000000001 2023-11-01T04:17:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:17:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138971+0000", 2023-11-01T04:17:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 58.454869369999997, 2023-11-01T04:17:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71733164100000002, 2023-11-01T04:17:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:17:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:17:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 116 2023-11-01T04:17:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138971+0000", 2023-11-01T04:17:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138971+0000", 2023-11-01T04:17:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138972+0000", 2023-11-01T04:17:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4000000000000001e-07 2023-11-01T04:17:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138988+0000", 2023-11-01T04:17:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6354999999999998e-05 2023-11-01T04:17:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138998+0000", 2023-11-01T04:17:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0285999999999999e-05 2023-11-01T04:17:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139010+0000", 2023-11-01T04:17:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1807000000000001e-05 2023-11-01T04:17:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.350520+0000", 2023-11-01T04:17:03.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21151004200000001 2023-11-01T04:17:03.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.350668+0000", 2023-11-01T04:17:03.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014824900000000001 2023-11-01T04:17:03.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:17:03.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.350873+0000", 2023-11-01T04:17:03.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020507899999999999 2023-11-01T04:17:03.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:17:03.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352744+0000", 2023-11-01T04:17:03.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001870741 2023-11-01T04:17:03.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:17:03.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856134+0000", 2023-11-01T04:17:03.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50339015399999998 2023-11-01T04:17:03.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:03.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856235+0000", 2023-11-01T04:17:03.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010087200000000001 2023-11-01T04:17:03.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856303+0000", 2023-11-01T04:17:03.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7316000000000004e-05 2023-11-01T04:17:03.640 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.641 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.641 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:17:03.642 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.139035+0000", 2023-11-01T04:17:03.642 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 58.454805383999997, 2023-11-01T04:17:03.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71770837499999995, 2023-11-01T04:17:03.642 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.642 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:03.642 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.643 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:17:03.643 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:17:03.643 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 117 2023-11-01T04:17:03.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.643 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139035+0000", 2023-11-01T04:17:03.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139035+0000", 2023-11-01T04:17:03.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139036+0000", 2023-11-01T04:17:03.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7999999999999995e-07 2023-11-01T04:17:03.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139042+0000", 2023-11-01T04:17:03.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9499999999999998e-06 2023-11-01T04:17:03.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139051+0000", 2023-11-01T04:17:03.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7489999999999993e-06 2023-11-01T04:17:03.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139059+0000", 2023-11-01T04:17:03.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8159999999999994e-06 2023-11-01T04:17:03.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.351436+0000", 2023-11-01T04:17:03.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21237641400000001 2023-11-01T04:17:03.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.351583+0000", 2023-11-01T04:17:03.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000147186 2023-11-01T04:17:03.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:17:03.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.351772+0000", 2023-11-01T04:17:03.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018879300000000001 2023-11-01T04:17:03.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:17:03.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.553963+0000", 2023-11-01T04:17:03.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20219130599999999 2023-11-01T04:17:03.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:17:03.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856458+0000", 2023-11-01T04:17:03.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30249546300000002 2023-11-01T04:17:03.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:03.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856632+0000", 2023-11-01T04:17:03.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000173611 2023-11-01T04:17:03.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856743+0000", 2023-11-01T04:17:03.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111407 2023-11-01T04:17:03.655 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.656 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.656 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.656 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:17:03.656 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622627+0000", 2023-11-01T04:17:03.657 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 56.971213685000002, 2023-11-01T04:17:03.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60704219400000003, 2023-11-01T04:17:03.657 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.657 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:03.657 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.657 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:17:03.658 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:17:03.658 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 139 2023-11-01T04:17:03.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.658 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622627+0000", 2023-11-01T04:17:03.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622627+0000", 2023-11-01T04:17:03.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622627+0000", 2023-11-01T04:17:03.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5400000000000001e-07 2023-11-01T04:17:03.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622632+0000", 2023-11-01T04:17:03.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0420000000000002e-06 2023-11-01T04:17:03.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622642+0000", 2023-11-01T04:17:03.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.713e-06 2023-11-01T04:17:03.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622649+0000", 2023-11-01T04:17:03.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4460000000000004e-06 2023-11-01T04:17:03.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.831442+0000", 2023-11-01T04:17:03.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20879259 2023-11-01T04:17:03.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.831689+0000", 2023-11-01T04:17:03.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000247087 2023-11-01T04:17:03.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:17:03.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.832051+0000", 2023-11-01T04:17:03.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036175699999999999 2023-11-01T04:17:03.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:17:03.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.837325+0000", 2023-11-01T04:17:03.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0052741289999999998 2023-11-01T04:17:03.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:17:03.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229409+0000", 2023-11-01T04:17:03.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39208416000000001 2023-11-01T04:17:03.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:03.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229566+0000", 2023-11-01T04:17:03.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015727499999999999 2023-11-01T04:17:03.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229669+0000", 2023-11-01T04:17:03.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000102341 2023-11-01T04:17:03.670 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.670 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.670 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.671 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:17:03.671 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622661+0000", 2023-11-01T04:17:03.671 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 56.971179292999999, 2023-11-01T04:17:03.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60735416200000003, 2023-11-01T04:17:03.672 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.672 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:03.672 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.672 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:17:03.672 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:17:03.673 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 140 2023-11-01T04:17:03.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.673 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622661+0000", 2023-11-01T04:17:03.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622661+0000", 2023-11-01T04:17:03.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622662+0000", 2023-11-01T04:17:03.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3e-07 2023-11-01T04:17:03.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622803+0000", 2023-11-01T04:17:03.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014142199999999999 2023-11-01T04:17:03.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622816+0000", 2023-11-01T04:17:03.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2683e-05 2023-11-01T04:17:03.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622825+0000", 2023-11-01T04:17:03.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9959999999999999e-06 2023-11-01T04:17:03.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.833052+0000", 2023-11-01T04:17:03.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21022711699999999 2023-11-01T04:17:03.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.833201+0000", 2023-11-01T04:17:03.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000149059 2023-11-01T04:17:03.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:17:03.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.833405+0000", 2023-11-01T04:17:03.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020363600000000001 2023-11-01T04:17:03.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:17:03.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.837343+0000", 2023-11-01T04:17:03.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039379590000000004 2023-11-01T04:17:03.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:17:03.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229871+0000", 2023-11-01T04:17:03.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39252865399999998 2023-11-01T04:17:03.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:03.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.229957+0000", 2023-11-01T04:17:03.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5962000000000001e-05 2023-11-01T04:17:03.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.230015+0000", 2023-11-01T04:17:03.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8044000000000002e-05 2023-11-01T04:17:03.685 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.685 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.685 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.686 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:17:03.686 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622843+0000", 2023-11-01T04:17:03.686 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 56.970996966999998, 2023-11-01T04:17:03.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70733220600000002, 2023-11-01T04:17:03.687 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.687 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:03.687 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.687 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:17:03.687 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:17:03.687 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 141 2023-11-01T04:17:03.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.688 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622843+0000", 2023-11-01T04:17:03.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622843+0000", 2023-11-01T04:17:03.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622844+0000", 2023-11-01T04:17:03.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7400000000000002e-07 2023-11-01T04:17:03.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622850+0000", 2023-11-01T04:17:03.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4330000000000003e-06 2023-11-01T04:17:03.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622859+0000", 2023-11-01T04:17:03.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5829999999999996e-06 2023-11-01T04:17:03.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.722958+0000", 2023-11-01T04:17:03.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100098824 2023-11-01T04:17:03.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835872+0000", 2023-11-01T04:17:03.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.112913586 2023-11-01T04:17:03.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.836107+0000", 2023-11-01T04:17:03.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023507399999999999 2023-11-01T04:17:03.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:17:03.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.836474+0000", 2023-11-01T04:17:03.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036769800000000002 2023-11-01T04:17:03.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:17:03.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.837993+0000", 2023-11-01T04:17:03.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001518192 2023-11-01T04:17:03.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:17:03.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.329913+0000", 2023-11-01T04:17:03.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.49192055499999998 2023-11-01T04:17:03.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:03.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.330067+0000", 2023-11-01T04:17:03.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000154012 2023-11-01T04:17:03.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.330176+0000", 2023-11-01T04:17:03.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010847499999999999 2023-11-01T04:17:03.700 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.700 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.700 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.701 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:207 3.2s0 3:42ba522a:::smithi19039307-20:head [copy-from ver 4] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:03.701 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.965579+0000", 2023-11-01T04:17:03.701 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.628260972, 2023-11-01T04:17:03.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62034388299999998, 2023-11-01T04:17:03.701 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.702 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:03.702 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.702 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:03.702 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:03.702 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 207 2023-11-01T04:17:03.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.703 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965579+0000", 2023-11-01T04:17:03.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965579+0000", 2023-11-01T04:17:03.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965581+0000", 2023-11-01T04:17:03.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1969999999999999e-06 2023-11-01T04:17:03.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965617+0000", 2023-11-01T04:17:03.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6100999999999999e-05 2023-11-01T04:17:03.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965640+0000", 2023-11-01T04:17:03.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3566000000000001e-05 2023-11-01T04:17:03.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965657+0000", 2023-11-01T04:17:03.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6413e-05 2023-11-01T04:17:03.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.066185+0000", 2023-11-01T04:17:03.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100528453 2023-11-01T04:17:03.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.066331+0000", 2023-11-01T04:17:03.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014583099999999999 2023-11-01T04:17:03.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:03.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.381227+0000", 2023-11-01T04:17:03.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31489625199999999 2023-11-01T04:17:03.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:03.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.384145+0000", 2023-11-01T04:17:03.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029175310000000001 2023-11-01T04:17:03.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:03.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.585865+0000", 2023-11-01T04:17:03.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.201720436 2023-11-01T04:17:03.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.585923+0000", 2023-11-01T04:17:03.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8103000000000003e-05 2023-11-01T04:17:03.717 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.717 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.717 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.718 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:209 3.3s0 3:ca2966a4:::smithi19039307-37:head [read 0~606208,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:03.718 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.966335+0000", 2023-11-01T04:17:03.718 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.627505112000001, 2023-11-01T04:17:03.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61218852199999996, 2023-11-01T04:17:03.718 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.719 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:03.719 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.719 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:03.719 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:03.719 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 209 2023-11-01T04:17:03.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.720 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966335+0000", 2023-11-01T04:17:03.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966335+0000", 2023-11-01T04:17:03.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966337+0000", 2023-11-01T04:17:03.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1430000000000001e-06 2023-11-01T04:17:03.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966349+0000", 2023-11-01T04:17:03.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2053e-05 2023-11-01T04:17:03.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966385+0000", 2023-11-01T04:17:03.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5438999999999997e-05 2023-11-01T04:17:03.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966408+0000", 2023-11-01T04:17:03.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3349e-05 2023-11-01T04:17:03.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970845+0000", 2023-11-01T04:17:03.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044366639999999999 2023-11-01T04:17:03.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970977+0000", 2023-11-01T04:17:03.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013261100000000001 2023-11-01T04:17:03.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.578524+0000", 2023-11-01T04:17:03.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60754626300000003 2023-11-01T04:17:03.728 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.728 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.728 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.729 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:210 3.3s0 3:ca2966a4:::smithi19039307-37:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:03.729 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.966473+0000", 2023-11-01T04:17:03.729 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.627367425000003, 2023-11-01T04:17:03.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61269195099999996, 2023-11-01T04:17:03.730 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.730 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:03.730 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.730 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:03.731 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:03.731 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 210 2023-11-01T04:17:03.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.731 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966473+0000", 2023-11-01T04:17:03.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966473+0000", 2023-11-01T04:17:03.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966474+0000", 2023-11-01T04:17:03.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4500000000000001e-06 2023-11-01T04:17:03.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.734 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966484+0000", 2023-11-01T04:17:03.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2269999999999999e-06 2023-11-01T04:17:03.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.735 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966508+0000", 2023-11-01T04:17:03.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4596e-05 2023-11-01T04:17:03.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966526+0000", 2023-11-01T04:17:03.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7938999999999999e-05 2023-11-01T04:17:03.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.071473+0000", 2023-11-01T04:17:03.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10494663799999999 2023-11-01T04:17:03.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.071640+0000", 2023-11-01T04:17:03.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016699 2023-11-01T04:17:03.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.579165+0000", 2023-11-01T04:17:03.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50752511099999997 2023-11-01T04:17:03.739 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.740 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.740 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.740 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:211 3.3s0 3:ca2966a4:::smithi19039307-37:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:03.740 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.966567+0000", 2023-11-01T04:17:03.741 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.627273787999997, 2023-11-01T04:17:03.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61304655900000005, 2023-11-01T04:17:03.741 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.741 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:03.741 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.742 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:03.742 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:03.742 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 211 2023-11-01T04:17:03.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.742 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966567+0000", 2023-11-01T04:17:03.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.744 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966567+0000", 2023-11-01T04:17:03.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966568+0000", 2023-11-01T04:17:03.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2529999999999999e-06 2023-11-01T04:17:03.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966575+0000", 2023-11-01T04:17:03.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4030000000000001e-06 2023-11-01T04:17:03.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966595+0000", 2023-11-01T04:17:03.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0208999999999999e-05 2023-11-01T04:17:03.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966611+0000", 2023-11-01T04:17:03.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6082999999999998e-05 2023-11-01T04:17:03.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.072213+0000", 2023-11-01T04:17:03.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105601509 2023-11-01T04:17:03.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.749 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.072432+0000", 2023-11-01T04:17:03.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000218724 2023-11-01T04:17:03.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.579613+0000", 2023-11-01T04:17:03.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50718137799999996 2023-11-01T04:17:03.751 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.751 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.751 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.752 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.752 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:212 3.3s0 3:c87735c0:::smithi19039307-50:head [create,setxattr XNeZS[lOPtVca\\Pa[\\rdfbinb[bcjhtYSpmcafZm (1928) in=40b,setxattr TfflYYqPnps]lo^qSlp___TitNdd`]kaPTVOTcsh (1648) in=40b,setxattr TNb_onpY[j_dkhfmnWtm]Yktg]]RNtbQe[OUTNlX (770) in=40b,setxattr ZcgnkhY\\orTYaZh]PPldj`SNq`p\\cX`k]h\\bZY_Y (700) in=40b,setxattr gPQVnqhpsVhW_kq`QnafXrlRZtZ\\YnRftgQTishR (595) in=40b,setxattr U`XqYi[WhoqnpSPRc\\RhnppZp^qY\\qUfjpqs[r^a (1892) in=40b,setxattr cTmhd`pWgrRf_VfVgOtgeOP^ShP]lsOtpoTj[\\gl (418) in=40b,setxattr aa_taYTfslg`lRRrOkkraN\\XiW^V\\ljl]NqXin`a (314) in=40b,setxattr ibl]`igtgVThY^Qse\\okYYUtTmolX\\W]p]Q[P\\il (1403) in=40b,setxattr \\ZabSO[UUqeZa`hddNh[ZjaRRgUZoQtkn\\QdsX]W (1541) in=40b,setxattr jUkQUPTNbo`rZNmYlUmhb]ojc]aeheoVRfp]bPWf (5052) in=40b,setxattr Ofj`oQ^cR]da^SPsTaXmmPse\\OZ`VSqQ]U`efsZl (419) in=40b,setxattr a[cdkWos\\OrUVPciUaa^_^`lWVTScffiWkd\\[kgf (1261) in=40b,setxattr jXb]\\WhbatdZ[qTTiZO]kPd^`l[kajOYiQ[iqSYc (244) in=40b,setxattr \\im]jQhbanUn\\_^h`jaU]W`ReaZN_je[]kgg (1578) in=36b,setxattr ZbosoRORS^_de]rN^_daOiha\\Q\\eRae^[sQUjXsV (9120) in=40b,setxattr gcn]XpZUkOZr\\pkrVVZpYiR\\alh_PaZa_S[PYcn] (384) in=40b,setxattr \\^qUOaNXSTjtmafhfbo\\gseTTkV`mgnacYq^bkcc (34) in=40b,setxattr iUa^mOQ]TqjVSUU_nXfY]f`NjljiiaYbrm[o]aes (1611) in=40b,setxattr tXaQPQbc^aiV[^reVOPpib^caoWf^XgqilO^_g[k (1016) in=40b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:03.752 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.970354+0000", 2023-11-01T04:17:03.752 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.623486178999997, 2023-11-01T04:17:03.752 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91502122100000005, 2023-11-01T04:17:03.753 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.753 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:03.753 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.753 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:03.753 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:03.753 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 212 2023-11-01T04:17:03.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.754 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.754 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.754 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970354+0000", 2023-11-01T04:17:03.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970354+0000", 2023-11-01T04:17:03.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.756 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.756 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970356+0000", 2023-11-01T04:17:03.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3070000000000001e-06 2023-11-01T04:17:03.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.757 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970468+0000", 2023-11-01T04:17:03.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111496 2023-11-01T04:17:03.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.758 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.758 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.758 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970492+0000", 2023-11-01T04:17:03.758 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4096000000000001e-05 2023-11-01T04:17:03.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.759 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.759 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.759 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970511+0000", 2023-11-01T04:17:03.759 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9069e-05 2023-11-01T04:17:03.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.760 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.760 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.760 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.579670+0000", 2023-11-01T04:17:03.760 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60915843300000005 2023-11-01T04:17:03.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.761 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.761 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.761 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.579779+0000", 2023-11-01T04:17:03.762 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000109417 2023-11-01T04:17:03.762 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.762 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.762 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:03.762 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.580741+0000", 2023-11-01T04:17:03.763 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00096157299999999996 2023-11-01T04:17:03.763 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.763 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.763 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:03.763 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.582542+0000", 2023-11-01T04:17:03.763 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018016130000000001 2023-11-01T04:17:03.764 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.764 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.764 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:03.764 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.885257+0000", 2023-11-01T04:17:03.765 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30271534300000003 2023-11-01T04:17:03.765 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.765 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.765 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.765 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.885375+0000", 2023-11-01T04:17:03.765 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117874 2023-11-01T04:17:03.766 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.766 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.766 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.766 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.767 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.767 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:496 3.2s0 3:4174a5c9:::smithi19039307-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2023-11-01T04:17:03.767 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:42.354747+0000", 2023-11-01T04:17:03.767 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 21.239093699000001, 2023-11-01T04:17:03.767 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61943324300000002, 2023-11-01T04:17:03.768 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.768 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:03.768 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.768 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:03.768 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:03.769 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 496 2023-11-01T04:17:03.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.769 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.769 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.769 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.770 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354747+0000", 2023-11-01T04:17:03.770 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.770 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.770 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.771 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354747+0000", 2023-11-01T04:17:03.771 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.771 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.771 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.771 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354747+0000", 2023-11-01T04:17:03.772 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0299999999999998e-07 2023-11-01T04:17:03.772 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.772 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.772 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.772 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354751+0000", 2023-11-01T04:17:03.773 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.326e-06 2023-11-01T04:17:03.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.773 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.773 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.773 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354763+0000", 2023-11-01T04:17:03.774 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2716e-05 2023-11-01T04:17:03.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.774 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.774 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.774 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354772+0000", 2023-11-01T04:17:03.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4570000000000005e-06 2023-11-01T04:17:03.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.775 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.775 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.775 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.357535+0000", 2023-11-01T04:17:03.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002763148 2023-11-01T04:17:03.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.776 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.776 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:03.776 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.357552+0000", 2023-11-01T04:17:03.777 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6849000000000001e-05 2023-11-01T04:17:03.777 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.777 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.777 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.777 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.568063+0000", 2023-11-01T04:17:03.777 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.210510788 2023-11-01T04:17:03.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.778 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.778 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:03.778 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.568200+0000", 2023-11-01T04:17:03.778 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013761000000000001 2023-11-01T04:17:03.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.779 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.779 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.779 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.670978+0000", 2023-11-01T04:17:03.779 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102777572 2023-11-01T04:17:03.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.780 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.780 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.780 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.671121+0000", 2023-11-01T04:17:03.780 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014276299999999999 2023-11-01T04:17:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.974180+0000", 2023-11-01T04:17:03.781 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.303059211 2023-11-01T04:17:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:552 3.2s0 3:42f00163:::smithi19039307-2:head [copy-from ver 12] snapc 1e=[1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.820369+0000", 2023-11-01T04:17:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 15.773471454999999, 2023-11-01T04:17:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89219467399999997, 2023-11-01T04:17:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.783 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:03.784 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.784 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:03.784 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:03.784 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 552 2023-11-01T04:17:03.784 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820369+0000", 2023-11-01T04:17:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.785 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820369+0000", 2023-11-01T04:17:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.786 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820371+0000", 2023-11-01T04:17:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.266e-06 2023-11-01T04:17:03.787 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820383+0000", 2023-11-01T04:17:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2023e-05 2023-11-01T04:17:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820415+0000", 2023-11-01T04:17:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2255999999999997e-05 2023-11-01T04:17:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.789 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820431+0000", 2023-11-01T04:17:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5823000000000001e-05 2023-11-01T04:17:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.790 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.079379+0000", 2023-11-01T04:17:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25894801099999998 2023-11-01T04:17:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.791 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.079550+0000", 2023-11-01T04:17:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017098600000000001 2023-11-01T04:17:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:03.793 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.494335+0000", 2023-11-01T04:17:03.793 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41478489200000002 2023-11-01T04:17:03.793 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.793 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.793 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:03.794 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.498143+0000", 2023-11-01T04:17:03.794 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038076920000000001 2023-11-01T04:17:03.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.794 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.794 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.712428+0000", 2023-11-01T04:17:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21428475699999999 2023-11-01T04:17:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.795 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.712564+0000", 2023-11-01T04:17:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013596800000000001 2023-11-01T04:17:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.796 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:553 3.2s0 3:42f00163:::smithi19039307-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.820458+0000", 2023-11-01T04:17:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 15.773382099999999, 2023-11-01T04:17:03.797 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.67673522200000003, 2023-11-01T04:17:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:03.798 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 553 2023-11-01T04:17:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.799 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820458+0000", 2023-11-01T04:17:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.800 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820458+0000", 2023-11-01T04:17:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.801 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820459+0000", 2023-11-01T04:17:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8299999999999995e-07 2023-11-01T04:17:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.802 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820463+0000", 2023-11-01T04:17:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9369999999999997e-06 2023-11-01T04:17:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820475+0000", 2023-11-01T04:17:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2026e-05 2023-11-01T04:17:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.804 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.920578+0000", 2023-11-01T04:17:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10010329 2023-11-01T04:17:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.805 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.283349+0000", 2023-11-01T04:17:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36277072599999999 2023-11-01T04:17:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for blocked object", 2023-11-01T04:17:03.806 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.283446+0000", 2023-11-01T04:17:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7083999999999994e-05 2023-11-01T04:17:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.496890+0000", 2023-11-01T04:17:03.807 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21344336899999999 2023-11-01T04:17:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.497036+0000", 2023-11-01T04:17:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000146324 2023-11-01T04:17:03.808 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.497193+0000", 2023-11-01T04:17:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015758300000000001 2023-11-01T04:17:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.809 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:565 3.2s0 3:42f00163:::smithi19039307-2:head [read 0~557056,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2023-11-01T04:17:03.810 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:49.511279+0000", 2023-11-01T04:17:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 14.082560822, 2023-11-01T04:17:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52628482899999995, 2023-11-01T04:17:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:03.811 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 565 2023-11-01T04:17:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.812 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511279+0000", 2023-11-01T04:17:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511279+0000", 2023-11-01T04:17:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.814 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511282+0000", 2023-11-01T04:17:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5969999999999999e-06 2023-11-01T04:17:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.815 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511294+0000", 2023-11-01T04:17:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1776e-05 2023-11-01T04:17:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.816 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511326+0000", 2023-11-01T04:17:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1686000000000001e-05 2023-11-01T04:17:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511341+0000", 2023-11-01T04:17:03.817 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5475999999999999e-05 2023-11-01T04:17:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511529+0000", 2023-11-01T04:17:03.818 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018760700000000001 2023-11-01T04:17:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511739+0000", 2023-11-01T04:17:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00021064 2023-11-01T04:17:03.819 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.037564+0000", 2023-11-01T04:17:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52582504699999999 2023-11-01T04:17:03.820 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:566 3.2s0 3:42f00163:::smithi19039307-2:head [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2023-11-01T04:17:03.821 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:49.511373+0000", 2023-11-01T04:17:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 14.082467380000001, 2023-11-01T04:17:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52738665100000004, 2023-11-01T04:17:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:03.822 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 566 2023-11-01T04:17:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511373+0000", 2023-11-01T04:17:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.824 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511373+0000", 2023-11-01T04:17:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.825 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511374+0000", 2023-11-01T04:17:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3170000000000001e-06 2023-11-01T04:17:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.826 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511381+0000", 2023-11-01T04:17:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.776e-06 2023-11-01T04:17:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.827 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511412+0000", 2023-11-01T04:17:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1093000000000002e-05 2023-11-01T04:17:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511431+0000", 2023-11-01T04:17:03.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8799000000000001e-05 2023-11-01T04:17:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.512308+0000", 2023-11-01T04:17:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00087712700000000001 2023-11-01T04:17:03.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.512476+0000", 2023-11-01T04:17:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016766700000000001 2023-11-01T04:17:03.830 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.038760+0000", 2023-11-01T04:17:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52628387200000004 2023-11-01T04:17:03.831 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:567 3.2s0 3:42f00163:::smithi19039307-2:head [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2023-11-01T04:17:03.832 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:49.511529+0000", 2023-11-01T04:17:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 14.082311475999999, 2023-11-01T04:17:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72495039999999999, 2023-11-01T04:17:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.833 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 567 2023-11-01T04:17:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511529+0000", 2023-11-01T04:17:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511529+0000", 2023-11-01T04:17:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.836 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511530+0000", 2023-11-01T04:17:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2759999999999999e-06 2023-11-01T04:17:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.837 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511537+0000", 2023-11-01T04:17:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2899999999999997e-06 2023-11-01T04:17:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511557+0000", 2023-11-01T04:17:03.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9443e-05 2023-11-01T04:17:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511577+0000", 2023-11-01T04:17:03.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9673e-05 2023-11-01T04:17:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.512998+0000", 2023-11-01T04:17:03.840 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014214360000000001 2023-11-01T04:17:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.513127+0000", 2023-11-01T04:17:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012906500000000001 2023-11-01T04:17:03.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.236479+0000", 2023-11-01T04:17:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72335221699999996 2023-11-01T04:17:03.842 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpReadReply(3.2s0 69/31 ECSubReadReply(tid=322, attrs_read=0))", 2023-11-01T04:17:03.843 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:49.514831+0000", 2023-11-01T04:17:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 14.079009052, 2023-11-01T04:17:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72027060200000004, 2023-11-01T04:17:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.844 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.514831+0000", 2023-11-01T04:17:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.845 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.514831+0000", 2023-11-01T04:17:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.846 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.514831+0000", 2023-11-01T04:17:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.847 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.514831+0000", 2023-11-01T04:17:03.848 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.848 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.848 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.848 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.848 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.614593+0000", 2023-11-01T04:17:03.848 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.099762016999999995 2023-11-01T04:17:03.849 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.849 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.849 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.849 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.714692+0000", 2023-11-01T04:17:03.849 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10009886599999999 2023-11-01T04:17:03.850 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.850 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.850 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.850 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.235030+0000", 2023-11-01T04:17:03.850 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52033755299999995 2023-11-01T04:17:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.235102+0000", 2023-11-01T04:17:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2166e-05 2023-11-01T04:17:03.851 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.852 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:571 3.2s0 3:4a13469b:::smithi19039307-24:head [setxattr _header (57) in=7b] snapc 22=[22,1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2023-11-01T04:17:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:49.541372+0000", 2023-11-01T04:17:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 14.052468705000001, 2023-11-01T04:17:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98667972400000004, 2023-11-01T04:17:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:03.853 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 571 2023-11-01T04:17:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.854 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541372+0000", 2023-11-01T04:17:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.855 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541372+0000", 2023-11-01T04:17:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.856 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541373+0000", 2023-11-01T04:17:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8300000000000001e-06 2023-11-01T04:17:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.857 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541386+0000", 2023-11-01T04:17:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2146e-05 2023-11-01T04:17:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.858 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541410+0000", 2023-11-01T04:17:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4672999999999999e-05 2023-11-01T04:17:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.859 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541427+0000", 2023-11-01T04:17:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6796e-05 2023-11-01T04:17:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:03.860 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.236577+0000", 2023-11-01T04:17:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.69514989800000004 2023-11-01T04:17:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:03.861 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.236756+0000", 2023-11-01T04:17:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000179211 2023-11-01T04:17:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:03.862 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.237530+0000", 2023-11-01T04:17:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077358099999999996 2023-11-01T04:17:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.238841+0000", 2023-11-01T04:17:03.863 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001311682 2023-11-01T04:17:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.527800+0000", 2023-11-01T04:17:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28895858200000002 2023-11-01T04:17:03.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.528051+0000", 2023-11-01T04:17:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000251325 2023-11-01T04:17:03.865 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:03.866 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:03.866 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:03.868 INFO:tasks.rados.rados.0.smithi190.stdout:290: finishing write tid 1 to smithi19039307-15 2023-11-01T04:17:03.868 INFO:tasks.rados.rados.0.smithi190.stdout:299: finishing write tid 1 to smithi19039307-31 2023-11-01T04:17:03.868 INFO:tasks.rados.rados.0.smithi190.stdout:294: expect (ObjNum 87 snap 20 seq_num 87) 2023-11-01T04:17:03.869 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-01T04:17:03.869 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:17:03.864+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:17:03.905 INFO:teuthology.orchestra.run.smithi153.stderr:nodeep-scrub is unset 2023-11-01T04:17:03.907 INFO:tasks.rados.rados.0.smithi190.stdout:290: finishing write tid 2 to smithi19039307-15 2023-11-01T04:17:03.907 INFO:tasks.rados.rados.0.smithi190.stdout:288: finishing copy_from racing read to smithi19039307-32 2023-11-01T04:17:03.911 INFO:tasks.rados.rados.0.smithi190.stdout:287: done (14 left) 2023-11-01T04:17:03.911 INFO:tasks.rados.rados.0.smithi190.stdout:294: done (13 left) 2023-11-01T04:17:03.911 INFO:tasks.rados.rados.0.smithi190.stdout:301: done (12 left) 2023-11-01T04:17:03.911 INFO:tasks.rados.rados.0.smithi190.stdout:302: rmattr oid 20 current snap is 26 2023-11-01T04:17:03.912 INFO:tasks.rados.rados.0.smithi190.stdout:302: done (12 left) 2023-11-01T04:17:03.912 INFO:tasks.rados.rados.0.smithi190.stdout:303: copy_from oid 42 from oid 19 current snap is 26 2023-11-01T04:17:03.912 INFO:tasks.rados.rados.0.smithi190.stdout:304: rollback oid 21 current snap is 26 2023-11-01T04:17:03.912 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 21 to 24 2023-11-01T04:17:03.912 INFO:tasks.rados.rados.0.smithi190.stdout:305: snap_remove snap 25 2023-11-01T04:17:03.969 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:03.966+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:04.007 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:17:04.007 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2023-11-01T04:17:04.007 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.007 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.5s5 83/31 ECSubWrite(tid=20, reqid=client.4559.0:744, at_version=83'20, trim_to=0'0, roll_forward_to=81'18))", 2023-11-01T04:17:04.008 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.917208+0000", 2023-11-01T04:17:04.008 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.090248483000000004, 2023-11-01T04:17:04.008 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.090293142000000007, 2023-11-01T04:17:04.008 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.008 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2023-11-01T04:17:04.008 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.009 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.009 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.009 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.917208+0000", 2023-11-01T04:17:04.010 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.010 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.010 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.010 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.011 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.917208+0000", 2023-11-01T04:17:04.011 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.011 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.011 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.011 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.012 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.917208+0000", 2023-11-01T04:17:04.012 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.43e-07 2023-11-01T04:17:04.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.012 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.012 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.013 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.917236+0000", 2023-11-01T04:17:04.013 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7421999999999999e-05 2023-11-01T04:17:04.013 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.013 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.013 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.013 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.917256+0000", 2023-11-01T04:17:04.014 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0767999999999998e-05 2023-11-01T04:17:04.014 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.014 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.014 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.014 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.917267+0000", 2023-11-01T04:17:04.015 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0706999999999999e-05 2023-11-01T04:17:04.015 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.015 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.015 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.015 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.016 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.016 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.5s5 83/31 ECSubWrite(tid=21, reqid=client.4559.0:745, at_version=83'21, trim_to=0'0, roll_forward_to=81'18))", 2023-11-01T04:17:04.016 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.920042+0000", 2023-11-01T04:17:04.016 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.087415050999999994, 2023-11-01T04:17:04.016 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.087545110999999995, 2023-11-01T04:17:04.017 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.017 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2023-11-01T04:17:04.017 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.017 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.017 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.017 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.920042+0000", 2023-11-01T04:17:04.018 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.018 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.018 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.920042+0000", 2023-11-01T04:17:04.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.019 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.019 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.019 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.019 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.920043+0000", 2023-11-01T04:17:04.020 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1179999999999999e-06 2023-11-01T04:17:04.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.020 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.020 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.020 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.920071+0000", 2023-11-01T04:17:04.021 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8317e-05 2023-11-01T04:17:04.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.021 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.021 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.021 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.920096+0000", 2023-11-01T04:17:04.022 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5194999999999998e-05 2023-11-01T04:17:04.022 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.022 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.022 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.022 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.920109+0000", 2023-11-01T04:17:04.023 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2584e-05 2023-11-01T04:17:04.023 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.023 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.023 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.024 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.024 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.5s5 83/31 ECSubWrite(tid=22, reqid=client.4559.0:746, at_version=83'22, trim_to=0'0, roll_forward_to=81'18))", 2023-11-01T04:17:04.024 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.921828+0000", 2023-11-01T04:17:04.024 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.085628179999999998, 2023-11-01T04:17:04.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.085833287999999994, 2023-11-01T04:17:04.024 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.025 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2023-11-01T04:17:04.025 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.025 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.025 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.025 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.921828+0000", 2023-11-01T04:17:04.025 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.026 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.026 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.026 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.026 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.921828+0000", 2023-11-01T04:17:04.026 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.027 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.027 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.027 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.027 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.921829+0000", 2023-11-01T04:17:04.028 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.049e-06 2023-11-01T04:17:04.028 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.028 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.028 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.028 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.921870+0000", 2023-11-01T04:17:04.028 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0085e-05 2023-11-01T04:17:04.029 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.029 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.029 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.029 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.921894+0000", 2023-11-01T04:17:04.029 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4618e-05 2023-11-01T04:17:04.029 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.030 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.921907+0000", 2023-11-01T04:17:04.030 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3302000000000001e-05 2023-11-01T04:17:04.031 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.031 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.031 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.031 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.031 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2023-11-01T04:17:04.031 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 3 2023-11-01T04:17:04.032 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:17:04.033 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-01T04:17:04.069 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:17:04.068+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:17:04.170 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:04.168+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:04.202 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:17:04.203 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:17:04.203 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2023-11-01T04:17:04.203 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2023-11-01T04:17:04.203 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:17:04.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-01T04:17:04.271 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:17:04.269+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:17:04.341 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:17:04.341 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2023-11-01T04:17:04.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2023-11-01T04:17:04.341 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2023-11-01T04:17:04.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.342 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.3s2 36/31 ECSubRead(tid=196, to_read={3:c87735c0:::smithi19039307-50:head=0,4096,0}, subchunks={3:c87735c0:::smithi19039307-50:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:17:04.342 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:15.821807+0000", 2023-11-01T04:17:04.342 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 48.516402952, 2023-11-01T04:17:04.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20089694399999999, 2023-11-01T04:17:04.343 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.343 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.343 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.821807+0000", 2023-11-01T04:17:04.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.344 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.821807+0000", 2023-11-01T04:17:04.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.345 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.345 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.345 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.345 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.821809+0000", 2023-11-01T04:17:04.345 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.296e-06 2023-11-01T04:17:04.346 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.346 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.346 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.346 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.821825+0000", 2023-11-01T04:17:04.346 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6463000000000001e-05 2023-11-01T04:17:04.346 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.347 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.347 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.347 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.821853+0000", 2023-11-01T04:17:04.347 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7512e-05 2023-11-01T04:17:04.347 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.347 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.348 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.348 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:15.921962+0000", 2023-11-01T04:17:04.348 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100109578 2023-11-01T04:17:04.348 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.348 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.349 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.349 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.022264+0000", 2023-11-01T04:17:04.349 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100301818 2023-11-01T04:17:04.349 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.349 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.349 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.022704+0000", 2023-11-01T04:17:04.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00044027699999999999 2023-11-01T04:17:04.350 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.350 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.350 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.350 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.351 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 37/31 ECSubWrite(tid=22, reqid=client.4559.0:187, at_version=37'15, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:17:04.351 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.969636+0000", 2023-11-01T04:17:04.351 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 47.368574361, 2023-11-01T04:17:04.351 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19731178799999999, 2023-11-01T04:17:04.352 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.352 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.352 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.352 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.352 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.352 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969636+0000", 2023-11-01T04:17:04.353 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.353 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.353 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.353 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.353 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969636+0000", 2023-11-01T04:17:04.353 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.354 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.354 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969638+0000", 2023-11-01T04:17:04.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9199999999999998e-06 2023-11-01T04:17:04.355 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.355 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.355 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.355 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969695+0000", 2023-11-01T04:17:04.355 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7497999999999997e-05 2023-11-01T04:17:04.355 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.356 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.356 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.356 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969721+0000", 2023-11-01T04:17:04.356 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5024999999999998e-05 2023-11-01T04:17:04.356 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.357 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.357 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.357 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.969736+0000", 2023-11-01T04:17:04.357 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5383000000000002e-05 2023-11-01T04:17:04.357 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.358 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.358 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.358 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.165457+0000", 2023-11-01T04:17:04.358 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19572152500000001 2023-11-01T04:17:04.358 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.359 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.359 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.359 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.165472+0000", 2023-11-01T04:17:04.359 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4713e-05 2023-11-01T04:17:04.359 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.360 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.360 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.360 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.166866+0000", 2023-11-01T04:17:04.360 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013936039999999999 2023-11-01T04:17:04.360 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.361 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.361 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.361 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.166948+0000", 2023-11-01T04:17:04.361 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2120000000000007e-05 2023-11-01T04:17:04.361 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.362 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.362 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.362 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.362 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.362 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 41/31 ECSubWrite(tid=58, reqid=osd.12.0:58, at_version=41'25, trim_to=0'0, roll_forward_to=37'19))", 2023-11-01T04:17:04.363 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:21.122902+0000", 2023-11-01T04:17:04.363 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 43.215308432999997, 2023-11-01T04:17:04.363 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105779886, 2023-11-01T04:17:04.363 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.364 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.364 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.364 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.364 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.364 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.122902+0000", 2023-11-01T04:17:04.365 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.365 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.365 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.365 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.365 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.122902+0000", 2023-11-01T04:17:04.365 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.366 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.366 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.366 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.366 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.122903+0000", 2023-11-01T04:17:04.366 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3370000000000001e-06 2023-11-01T04:17:04.367 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.367 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.367 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.367 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.122949+0000", 2023-11-01T04:17:04.367 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5649000000000003e-05 2023-11-01T04:17:04.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.368 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.368 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.368 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.122981+0000", 2023-11-01T04:17:04.368 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2502000000000002e-05 2023-11-01T04:17:04.368 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.369 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.369 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.369 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.223072+0000", 2023-11-01T04:17:04.369 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100090422 2023-11-01T04:17:04.369 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.370 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.370 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.370 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.223448+0000", 2023-11-01T04:17:04.370 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00037613399999999997 2023-11-01T04:17:04.370 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.371 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.371 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.371 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.223471+0000", 2023-11-01T04:17:04.371 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3088999999999999e-05 2023-11-01T04:17:04.371 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.372 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.372 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.372 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.228640+0000", 2023-11-01T04:17:04.372 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051686800000000001 2023-11-01T04:17:04.372 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.373 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.373 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.373 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:21.228682+0000", 2023-11-01T04:17:04.373 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2073000000000002e-05 2023-11-01T04:17:04.373 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.373 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.374 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.374 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.374 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 53/31 ECSubWrite(tid=82, reqid=client.4559.0:405, at_version=53'37, trim_to=0'0, roll_forward_to=50'35))", 2023-11-01T04:17:04.374 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:33.313117+0000", 2023-11-01T04:17:04.375 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 31.025093128999998, 2023-11-01T04:17:04.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104799267, 2023-11-01T04:17:04.375 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.375 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.375 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313117+0000", 2023-11-01T04:17:04.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313117+0000", 2023-11-01T04:17:04.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.378 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.378 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313118+0000", 2023-11-01T04:17:04.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9999999999999998e-07 2023-11-01T04:17:04.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.379 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.425 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313430+0000", 2023-11-01T04:17:04.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00031231199999999997 2023-11-01T04:17:04.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.426 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.426 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.426 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313460+0000", 2023-11-01T04:17:04.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9879999999999999e-05 2023-11-01T04:17:04.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313477+0000", 2023-11-01T04:17:04.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7186000000000001e-05 2023-11-01T04:17:04.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313619+0000", 2023-11-01T04:17:04.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014189699999999999 2023-11-01T04:17:04.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.313648+0000", 2023-11-01T04:17:04.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8812000000000001e-05 2023-11-01T04:17:04.430 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.417809+0000", 2023-11-01T04:17:04.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104160975 2023-11-01T04:17:04.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.417917+0000", 2023-11-01T04:17:04.432 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000107705 2023-11-01T04:17:04.432 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.432 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.432 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.433 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 57/31 ECSubWrite(tid=104, reqid=osd.12.0:104, at_version=57'56, trim_to=0'0, roll_forward_to=55'52))", 2023-11-01T04:17:04.433 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:37.237714+0000", 2023-11-01T04:17:04.433 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 27.100496837000001, 2023-11-01T04:17:04.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104091738, 2023-11-01T04:17:04.434 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.434 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.434 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.237714+0000", 2023-11-01T04:17:04.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.237714+0000", 2023-11-01T04:17:04.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.237715+0000", 2023-11-01T04:17:04.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.223e-06 2023-11-01T04:17:04.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.438 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.237765+0000", 2023-11-01T04:17:04.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9771999999999997e-05 2023-11-01T04:17:04.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.439 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.237798+0000", 2023-11-01T04:17:04.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3732999999999997e-05 2023-11-01T04:17:04.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.439 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.237822+0000", 2023-11-01T04:17:04.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3181999999999999e-05 2023-11-01T04:17:04.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.338057+0000", 2023-11-01T04:17:04.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10023538999999999 2023-11-01T04:17:04.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.338079+0000", 2023-11-01T04:17:04.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1820000000000001e-05 2023-11-01T04:17:04.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.341719+0000", 2023-11-01T04:17:04.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003640407 2023-11-01T04:17:04.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:37.341805+0000", 2023-11-01T04:17:04.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6211000000000005e-05 2023-11-01T04:17:04.444 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.444 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.444 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.445 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 67/31 ECSubWrite(tid=77, reqid=client.4559.0:537, at_version=67'58, trim_to=0'0, roll_forward_to=67'56))", 2023-11-01T04:17:04.445 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:47.705575+0000", 2023-11-01T04:17:04.445 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.632635338, 2023-11-01T04:17:04.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.198004336, 2023-11-01T04:17:04.446 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.446 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.446 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.705575+0000", 2023-11-01T04:17:04.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.705575+0000", 2023-11-01T04:17:04.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.705576+0000", 2023-11-01T04:17:04.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6199999999999996e-07 2023-11-01T04:17:04.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.706201+0000", 2023-11-01T04:17:04.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00062493599999999998 2023-11-01T04:17:04.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.706232+0000", 2023-11-01T04:17:04.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0766000000000003e-05 2023-11-01T04:17:04.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.706256+0000", 2023-11-01T04:17:04.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4358999999999998e-05 2023-11-01T04:17:04.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.452 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.452 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.802580+0000", 2023-11-01T04:17:04.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.096323874000000004 2023-11-01T04:17:04.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.802593+0000", 2023-11-01T04:17:04.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2634e-05 2023-11-01T04:17:04.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.903487+0000", 2023-11-01T04:17:04.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10089468 2023-11-01T04:17:04.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.903579+0000", 2023-11-01T04:17:04.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2225000000000006e-05 2023-11-01T04:17:04.455 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.455 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.456 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.456 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s4 69/31 ECSubWrite(tid=85, reqid=client.4559.0:555, at_version=69'46, trim_to=0'0, roll_forward_to=68'44))", 2023-11-01T04:17:04.456 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:48.790183+0000", 2023-11-01T04:17:04.456 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.548026948, 2023-11-01T04:17:04.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20311115800000001, 2023-11-01T04:17:04.457 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.457 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.457 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.790183+0000", 2023-11-01T04:17:04.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.790183+0000", 2023-11-01T04:17:04.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.790185+0000", 2023-11-01T04:17:04.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4729999999999999e-06 2023-11-01T04:17:04.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.790243+0000", 2023-11-01T04:17:04.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7843000000000002e-05 2023-11-01T04:17:04.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.790282+0000", 2023-11-01T04:17:04.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9255999999999998e-05 2023-11-01T04:17:04.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.890377+0000", 2023-11-01T04:17:04.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100094456 2023-11-01T04:17:04.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.990579+0000", 2023-11-01T04:17:04.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100202728 2023-11-01T04:17:04.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.990596+0000", 2023-11-01T04:17:04.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6906999999999999e-05 2023-11-01T04:17:04.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.993193+0000", 2023-11-01T04:17:04.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025967630000000002 2023-11-01T04:17:04.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.993295+0000", 2023-11-01T04:17:04.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010173199999999999 2023-11-01T04:17:04.467 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.467 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.468 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.468 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 69/31 ECSubWrite(tid=82, reqid=client.4559.0:561, at_version=69'64, trim_to=0'0, roll_forward_to=68'62))", 2023-11-01T04:17:04.468 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:49.711973+0000", 2023-11-01T04:17:04.469 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 14.626237019, 2023-11-01T04:17:04.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105150903, 2023-11-01T04:17:04.469 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.469 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.469 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.711973+0000", 2023-11-01T04:17:04.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.711973+0000", 2023-11-01T04:17:04.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.472 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.711975+0000", 2023-11-01T04:17:04.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9400000000000001e-06 2023-11-01T04:17:04.472 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.472 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.473 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.473 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712340+0000", 2023-11-01T04:17:04.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00036447999999999998 2023-11-01T04:17:04.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.473 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712372+0000", 2023-11-01T04:17:04.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2113000000000002e-05 2023-11-01T04:17:04.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712388+0000", 2023-11-01T04:17:04.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5835999999999999e-05 2023-11-01T04:17:04.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712536+0000", 2023-11-01T04:17:04.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014780800000000001 2023-11-01T04:17:04.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.476 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712563+0000", 2023-11-01T04:17:04.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6951999999999999e-05 2023-11-01T04:17:04.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.477 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.817062+0000", 2023-11-01T04:17:04.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104499095 2023-11-01T04:17:04.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.817124+0000", 2023-11-01T04:17:04.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2679000000000001e-05 2023-11-01T04:17:04.478 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.478 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.479 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.479 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 69/31 ECSubWrite(tid=83, reqid=client.4559.0:562, at_version=69'65, trim_to=0'0, roll_forward_to=68'62))", 2023-11-01T04:17:04.479 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:49.712424+0000", 2023-11-01T04:17:04.480 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 14.625786616999999, 2023-11-01T04:17:04.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104741764, 2023-11-01T04:17:04.480 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.480 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.480 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.481 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.481 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712424+0000", 2023-11-01T04:17:04.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.482 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.482 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712424+0000", 2023-11-01T04:17:04.482 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.482 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712425+0000", 2023-11-01T04:17:04.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9299999999999997e-07 2023-11-01T04:17:04.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.484 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.484 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712599+0000", 2023-11-01T04:17:04.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017441700000000001 2023-11-01T04:17:04.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.484 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.485 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.485 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712615+0000", 2023-11-01T04:17:04.485 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6120000000000002e-05 2023-11-01T04:17:04.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712628+0000", 2023-11-01T04:17:04.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3223e-05 2023-11-01T04:17:04.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.713736+0000", 2023-11-01T04:17:04.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011081660000000001 2023-11-01T04:17:04.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.488 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.488 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.713764+0000", 2023-11-01T04:17:04.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7708999999999998e-05 2023-11-01T04:17:04.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.488 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.817134+0000", 2023-11-01T04:17:04.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103369669 2023-11-01T04:17:04.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.489 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.490 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.817166+0000", 2023-11-01T04:17:04.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1767000000000003e-05 2023-11-01T04:17:04.490 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.490 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.490 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.491 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.491 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.491 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 69/31 ECSubWrite(tid=84, reqid=client.4559.0:563, at_version=69'66, trim_to=0'0, roll_forward_to=68'62))", 2023-11-01T04:17:04.491 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:49.712651+0000", 2023-11-01T04:17:04.491 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 14.625559075, 2023-11-01T04:17:04.491 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10454166400000001, 2023-11-01T04:17:04.492 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.492 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.492 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712651+0000", 2023-11-01T04:17:04.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.493 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.493 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712651+0000", 2023-11-01T04:17:04.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.494 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.494 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712652+0000", 2023-11-01T04:17:04.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3300000000000002e-07 2023-11-01T04:17:04.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712671+0000", 2023-11-01T04:17:04.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9174e-05 2023-11-01T04:17:04.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.496 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.496 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.496 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712684+0000", 2023-11-01T04:17:04.496 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2723e-05 2023-11-01T04:17:04.496 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.496 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.497 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.712705+0000", 2023-11-01T04:17:04.497 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1229e-05 2023-11-01T04:17:04.497 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.498 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.815270+0000", 2023-11-01T04:17:04.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102564498 2023-11-01T04:17:04.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.498 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.498 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.499 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.815286+0000", 2023-11-01T04:17:04.499 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5903e-05 2023-11-01T04:17:04.499 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.499 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.499 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.499 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.817168+0000", 2023-11-01T04:17:04.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001882512 2023-11-01T04:17:04.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.500 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.500 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.500 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.817193+0000", 2023-11-01T04:17:04.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4992000000000001e-05 2023-11-01T04:17:04.501 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.501 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.501 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.501 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 69/31 ECSubWrite(tid=85, reqid=client.4559.0:573, at_version=69'68, trim_to=0'0, roll_forward_to=69'63))", 2023-11-01T04:17:04.502 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:49.714753+0000", 2023-11-01T04:17:04.502 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 14.623457581, 2023-11-01T04:17:04.502 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10370937600000001, 2023-11-01T04:17:04.502 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.502 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.503 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.503 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.503 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.714753+0000", 2023-11-01T04:17:04.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.504 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.714753+0000", 2023-11-01T04:17:04.504 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.504 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.504 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.505 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.505 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.714755+0000", 2023-11-01T04:17:04.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7e-06 2023-11-01T04:17:04.505 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.505 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.714801+0000", 2023-11-01T04:17:04.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6415999999999997e-05 2023-11-01T04:17:04.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.714833+0000", 2023-11-01T04:17:04.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1659999999999998e-05 2023-11-01T04:17:04.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.714849+0000", 2023-11-01T04:17:04.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6008999999999999e-05 2023-11-01T04:17:04.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.508 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.815789+0000", 2023-11-01T04:17:04.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100940838 2023-11-01T04:17:04.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.509 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.815810+0000", 2023-11-01T04:17:04.509 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0100000000000001e-05 2023-11-01T04:17:04.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.510 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.818396+0000", 2023-11-01T04:17:04.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002586428 2023-11-01T04:17:04.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.818462+0000", 2023-11-01T04:17:04.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6224999999999997e-05 2023-11-01T04:17:04.511 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.511 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.512 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.512 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s4 73/72 ECSubWrite(tid=107, reqid=client.4559.0:611, at_version=73'62, trim_to=0'0, roll_forward_to=0'0))", 2023-11-01T04:17:04.512 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:54.133863+0000", 2023-11-01T04:17:04.513 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.204347840000001, 2023-11-01T04:17:04.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.104046954, 2023-11-01T04:17:04.513 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.513 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.513 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.514 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.514 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.133863+0000", 2023-11-01T04:17:04.514 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.514 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.514 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.133863+0000", 2023-11-01T04:17:04.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.133865+0000", 2023-11-01T04:17:04.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0159999999999998e-06 2023-11-01T04:17:04.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.134252+0000", 2023-11-01T04:17:04.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00038782399999999999 2023-11-01T04:17:04.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.134299+0000", 2023-11-01T04:17:04.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6816e-05 2023-11-01T04:17:04.518 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.518 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.518 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.234403+0000", 2023-11-01T04:17:04.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10010385300000001 2023-11-01T04:17:04.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.234573+0000", 2023-11-01T04:17:04.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000169617 2023-11-01T04:17:04.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.234607+0000", 2023-11-01T04:17:04.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4233000000000002e-05 2023-11-01T04:17:04.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.521 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.237826+0000", 2023-11-01T04:17:04.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032187829999999998 2023-11-01T04:17:04.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.522 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.522 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.237910+0000", 2023-11-01T04:17:04.522 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3812000000000003e-05 2023-11-01T04:17:04.522 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.522 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.523 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.523 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.fs6 79/31 ECSubWrite(tid=41, reqid=osd.0.0:41, at_version=79'24, trim_to=0'0, roll_forward_to=76'22))", 2023-11-01T04:17:04.523 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:00.070897+0000", 2023-11-01T04:17:04.524 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.2673133319999996, 2023-11-01T04:17:04.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110177715, 2023-11-01T04:17:04.524 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.524 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.524 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.070897+0000", 2023-11-01T04:17:04.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.526 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.070897+0000", 2023-11-01T04:17:04.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.526 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.527 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.070899+0000", 2023-11-01T04:17:04.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.449e-06 2023-11-01T04:17:04.527 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.527 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.070957+0000", 2023-11-01T04:17:04.528 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8276000000000002e-05 2023-11-01T04:17:04.528 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.528 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.528 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.070983+0000", 2023-11-01T04:17:04.529 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5678e-05 2023-11-01T04:17:04.529 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.529 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.529 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.529 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.070997+0000", 2023-11-01T04:17:04.529 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4688000000000001e-05 2023-11-01T04:17:04.530 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.530 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.530 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.530 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.171285+0000", 2023-11-01T04:17:04.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100288021 2023-11-01T04:17:04.531 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.531 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.531 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.171303+0000", 2023-11-01T04:17:04.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7750000000000001e-05 2023-11-01T04:17:04.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.180974+0000", 2023-11-01T04:17:04.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0096710560000000008 2023-11-01T04:17:04.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.181075+0000", 2023-11-01T04:17:04.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000100797 2023-11-01T04:17:04.533 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.533 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.534 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.534 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.534 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.534 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 79/73 ECSubWrite(tid=133, reqid=osd.11.0:132, at_version=0'0, trim_to=0'0, roll_forward_to=79'124))", 2023-11-01T04:17:04.534 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:00.390398+0000", 2023-11-01T04:17:04.534 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.947812581, 2023-11-01T04:17:04.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.205429638, 2023-11-01T04:17:04.535 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.535 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.535 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.390398+0000", 2023-11-01T04:17:04.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.390398+0000", 2023-11-01T04:17:04.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.390400+0000", 2023-11-01T04:17:04.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9700000000000002e-06 2023-11-01T04:17:04.538 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.538 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.538 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.538 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.390442+0000", 2023-11-01T04:17:04.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1947999999999999e-05 2023-11-01T04:17:04.539 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.539 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.539 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.539 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.390476+0000", 2023-11-01T04:17:04.539 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4669000000000003e-05 2023-11-01T04:17:04.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.540 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.490565+0000", 2023-11-01T04:17:04.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100088395 2023-11-01T04:17:04.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.541 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.490683+0000", 2023-11-01T04:17:04.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011819699999999999 2023-11-01T04:17:04.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.542 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.542 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.490705+0000", 2023-11-01T04:17:04.542 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1588999999999999e-05 2023-11-01T04:17:04.542 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.595766+0000", 2023-11-01T04:17:04.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105061275 2023-11-01T04:17:04.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.544 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.544 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.595827+0000", 2023-11-01T04:17:04.544 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1594999999999996e-05 2023-11-01T04:17:04.544 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.544 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.544 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.545 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.545 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.545 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 79/73 ECSubWrite(tid=134, reqid=osd.11.0:134, at_version=79'126, trim_to=0'0, roll_forward_to=79'124))", 2023-11-01T04:17:04.545 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:00.490610+0000", 2023-11-01T04:17:04.545 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.8476000539999999, 2023-11-01T04:17:04.545 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105789521, 2023-11-01T04:17:04.546 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.546 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.546 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.490610+0000", 2023-11-01T04:17:04.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.490610+0000", 2023-11-01T04:17:04.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.548 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.548 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.490612+0000", 2023-11-01T04:17:04.548 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6500000000000001e-06 2023-11-01T04:17:04.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.549 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.549 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.549 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.490686+0000", 2023-11-01T04:17:04.549 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4288000000000004e-05 2023-11-01T04:17:04.549 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.549 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.550 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.550 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.490707+0000", 2023-11-01T04:17:04.550 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0326000000000002e-05 2023-11-01T04:17:04.550 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.490729+0000", 2023-11-01T04:17:04.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2819e-05 2023-11-01T04:17:04.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.552 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.593567+0000", 2023-11-01T04:17:04.552 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102837778 2023-11-01T04:17:04.552 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.552 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.552 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.553 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.593588+0000", 2023-11-01T04:17:04.553 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0806e-05 2023-11-01T04:17:04.553 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.553 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.553 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.553 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.596353+0000", 2023-11-01T04:17:04.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027648009999999999 2023-11-01T04:17:04.554 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.555 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.596400+0000", 2023-11-01T04:17:04.555 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7052999999999998e-05 2023-11-01T04:17:04.555 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.555 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.555 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.556 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.556 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.es4 79/73 ECSubWrite(tid=135, reqid=osd.11.0:135, at_version=79'128, trim_to=0'0, roll_forward_to=79'124))", 2023-11-01T04:17:04.556 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:00.493205+0000", 2023-11-01T04:17:04.556 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.8450051809999999, 2023-11-01T04:17:04.556 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10435291300000001, 2023-11-01T04:17:04.556 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.557 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.557 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.557 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.557 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.557 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.493205+0000", 2023-11-01T04:17:04.558 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.558 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.558 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.558 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.558 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.493205+0000", 2023-11-01T04:17:04.558 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.559 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.559 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.559 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.559 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.493207+0000", 2023-11-01T04:17:04.559 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.68e-06 2023-11-01T04:17:04.560 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.560 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.560 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.560 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.493253+0000", 2023-11-01T04:17:04.561 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6346e-05 2023-11-01T04:17:04.561 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.561 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.561 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.561 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.493282+0000", 2023-11-01T04:17:04.562 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8782e-05 2023-11-01T04:17:04.562 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.562 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.562 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.493298+0000", 2023-11-01T04:17:04.563 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5492999999999999e-05 2023-11-01T04:17:04.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.563 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.563 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.594473+0000", 2023-11-01T04:17:04.564 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10117571 2023-11-01T04:17:04.564 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.564 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.564 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.564 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.594492+0000", 2023-11-01T04:17:04.564 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9162999999999999e-05 2023-11-01T04:17:04.565 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.565 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.565 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.565 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.597481+0000", 2023-11-01T04:17:04.565 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002988901 2023-11-01T04:17:04.566 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.566 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.566 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.566 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.597558+0000", 2023-11-01T04:17:04.566 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6837999999999999e-05 2023-11-01T04:17:04.566 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.567 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.567 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.567 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.567 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.567 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds5 82/68 ECSubWrite(tid=35, reqid=client.4559.0:723, at_version=82'97, trim_to=0'0, roll_forward_to=79'94))", 2023-11-01T04:17:04.567 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.736403+0000", 2023-11-01T04:17:04.568 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.60180741000000004, 2023-11-01T04:17:04.568 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10439611, 2023-11-01T04:17:04.568 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.568 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.568 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.568 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.569 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.569 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.736403+0000", 2023-11-01T04:17:04.569 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.569 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.569 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.569 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.570 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.736403+0000", 2023-11-01T04:17:04.570 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.570 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.570 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.570 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.571 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.736404+0000", 2023-11-01T04:17:04.571 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1299999999999999e-07 2023-11-01T04:17:04.571 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.571 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.571 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.571 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.736469+0000", 2023-11-01T04:17:04.572 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5252000000000005e-05 2023-11-01T04:17:04.572 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.572 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.572 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.572 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.736493+0000", 2023-11-01T04:17:04.572 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4406e-05 2023-11-01T04:17:04.573 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.573 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.573 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.573 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.836608+0000", 2023-11-01T04:17:04.573 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100114671 2023-11-01T04:17:04.573 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.574 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.574 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.574 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.836797+0000", 2023-11-01T04:17:04.574 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000188729 2023-11-01T04:17:04.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.575 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.575 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.575 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.836812+0000", 2023-11-01T04:17:04.575 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5299e-05 2023-11-01T04:17:04.575 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.576 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.576 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.576 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.840709+0000", 2023-11-01T04:17:04.576 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038966560000000001 2023-11-01T04:17:04.576 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.576 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.577 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.577 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.840799+0000", 2023-11-01T04:17:04.577 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0284000000000005e-05 2023-11-01T04:17:04.577 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.577 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.577 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.578 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.578 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.578 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.5s5 83/31 ECSubWrite(tid=20, reqid=client.4559.0:744, at_version=83'20, trim_to=0'0, roll_forward_to=81'18))", 2023-11-01T04:17:04.578 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.917208+0000", 2023-11-01T04:17:04.578 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.42100230500000002, 2023-11-01T04:17:04.579 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20240476800000001, 2023-11-01T04:17:04.579 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.579 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.579 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.579 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.579 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.580 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.917208+0000", 2023-11-01T04:17:04.580 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.580 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.580 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.580 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.917208+0000", 2023-11-01T04:17:04.581 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.581 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.581 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.581 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.581 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.917208+0000", 2023-11-01T04:17:04.581 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.43e-07 2023-11-01T04:17:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.917236+0000", 2023-11-01T04:17:04.582 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7421999999999999e-05 2023-11-01T04:17:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.917256+0000", 2023-11-01T04:17:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0767999999999998e-05 2023-11-01T04:17:04.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.917267+0000", 2023-11-01T04:17:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0706999999999999e-05 2023-11-01T04:17:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.017510+0000", 2023-11-01T04:17:04.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100243318 2023-11-01T04:17:04.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.017529+0000", 2023-11-01T04:17:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8938999999999999e-05 2023-11-01T04:17:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.586 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.119552+0000", 2023-11-01T04:17:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102022874 2023-11-01T04:17:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.587 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.119613+0000", 2023-11-01T04:17:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0597000000000003e-05 2023-11-01T04:17:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.588 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.5s5 83/31 ECSubWrite(tid=21, reqid=client.4559.0:745, at_version=83'21, trim_to=0'0, roll_forward_to=81'18))", 2023-11-01T04:17:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.920042+0000", 2023-11-01T04:17:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.418168873, 2023-11-01T04:17:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19961890900000001, 2023-11-01T04:17:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.589 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.590 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.590 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.590 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.590 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.920042+0000", 2023-11-01T04:17:04.590 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.591 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.591 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.591 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.591 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.920042+0000", 2023-11-01T04:17:04.591 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.920043+0000", 2023-11-01T04:17:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1179999999999999e-06 2023-11-01T04:17:04.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.920071+0000", 2023-11-01T04:17:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8317e-05 2023-11-01T04:17:04.593 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.594 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.594 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.594 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.920096+0000", 2023-11-01T04:17:04.594 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5194999999999998e-05 2023-11-01T04:17:04.594 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.595 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.595 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.595 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.920109+0000", 2023-11-01T04:17:04.595 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2584e-05 2023-11-01T04:17:04.595 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.596 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.596 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.596 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.019350+0000", 2023-11-01T04:17:04.596 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099241423999999995 2023-11-01T04:17:04.596 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.596 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.597 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.597 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.019367+0000", 2023-11-01T04:17:04.597 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7062e-05 2023-11-01T04:17:04.597 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.597 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.597 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.598 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.119620+0000", 2023-11-01T04:17:04.598 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100252893 2023-11-01T04:17:04.598 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.598 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.598 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.599 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.119660+0000", 2023-11-01T04:17:04.599 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0315999999999998e-05 2023-11-01T04:17:04.599 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.599 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.599 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.600 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.600 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.5s5 83/31 ECSubWrite(tid=22, reqid=client.4559.0:746, at_version=83'22, trim_to=0'0, roll_forward_to=81'18))", 2023-11-01T04:17:04.600 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.921828+0000", 2023-11-01T04:17:04.600 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.41638200199999997, 2023-11-01T04:17:04.600 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19893837, 2023-11-01T04:17:04.601 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:04.601 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:04.601 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:04.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.601 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:04.601 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.921828+0000", 2023-11-01T04:17:04.602 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.602 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.602 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.602 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:04.602 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.921828+0000", 2023-11-01T04:17:04.603 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:04.603 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.603 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.603 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:04.603 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.921829+0000", 2023-11-01T04:17:04.604 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.049e-06 2023-11-01T04:17:04.604 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.604 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.604 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:04.604 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.921870+0000", 2023-11-01T04:17:04.604 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0085e-05 2023-11-01T04:17:04.605 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.605 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:04.605 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.921894+0000", 2023-11-01T04:17:04.605 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4618e-05 2023-11-01T04:17:04.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.606 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.606 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:04.606 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.921907+0000", 2023-11-01T04:17:04.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3302000000000001e-05 2023-11-01T04:17:04.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.607 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.607 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:04.607 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.119281+0000", 2023-11-01T04:17:04.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19737337299999999 2023-11-01T04:17:04.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.608 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:04.608 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.119293+0000", 2023-11-01T04:17:04.608 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1701e-05 2023-11-01T04:17:04.608 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.609 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:04.609 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.120708+0000", 2023-11-01T04:17:04.609 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001415101 2023-11-01T04:17:04.609 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:04.609 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:04.610 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:04.610 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.120767+0000", 2023-11-01T04:17:04.610 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9141000000000002e-05 2023-11-01T04:17:04.610 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.610 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.611 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.611 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:04.611 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:04.611 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:17:04.612 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-01T04:17:04.613 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:17:04.610+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:17:04.714 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:04.712+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:04.789 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:17:04.789 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:17:04.789 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2023-11-01T04:17:04.789 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:17:04.803 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-01T04:17:04.848 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:17:04.846+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:17:04.914 INFO:tasks.rados.rados.0.smithi190.stdout:305: done (14 left) 2023-11-01T04:17:04.914 INFO:tasks.rados.rados.0.smithi190.stdout:306: snap_remove snap 20 2023-11-01T04:17:04.915 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:17:04.913+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:17:04.925 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:17:04.925 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:17:04.926 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2023-11-01T04:17:04.926 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2023-11-01T04:17:04.926 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:17:04.939 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-01T04:17:05.028 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:05.026+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:05.108 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:17:05.109 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2023-11-01T04:17:05.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2023-11-01T04:17:05.109 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2023-11-01T04:17:05.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.109 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:357 3.1s0 3:8900b4ea:::smithi19039307-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:17:05.110 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:27.480248+0000", 2023-11-01T04:17:05.110 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 37.625900219000002, 2023-11-01T04:17:05.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41393773900000003, 2023-11-01T04:17:05.110 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.110 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:05.111 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.111 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.111 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.111 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 357 2023-11-01T04:17:05.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.111 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480248+0000", 2023-11-01T04:17:05.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480248+0000", 2023-11-01T04:17:05.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480249+0000", 2023-11-01T04:17:05.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.16e-07 2023-11-01T04:17:05.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480253+0000", 2023-11-01T04:17:05.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7270000000000001e-06 2023-11-01T04:17:05.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480264+0000", 2023-11-01T04:17:05.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0693000000000001e-05 2023-11-01T04:17:05.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.480272+0000", 2023-11-01T04:17:05.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0059999999999992e-06 2023-11-01T04:17:05.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.688898+0000", 2023-11-01T04:17:05.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20862633899999999 2023-11-01T04:17:05.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:05.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.689066+0000", 2023-11-01T04:17:05.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016846499999999999 2023-11-01T04:17:05.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.691994+0000", 2023-11-01T04:17:05.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029271100000000001 2023-11-01T04:17:05.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.692157+0000", 2023-11-01T04:17:05.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016358799999999999 2023-11-01T04:17:05.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:27.894186+0000", 2023-11-01T04:17:05.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20202899499999999 2023-11-01T04:17:05.122 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.122 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.122 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.123 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:505 3.1s0 3:8900b4ea:::smithi19039307-31:head [setxattr _header (57) in=7b] snapc 1d=[1d,1c,1b,19,18,15,12,11,b,4] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2023-11-01T04:17:05.123 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:44.244582+0000", 2023-11-01T04:17:05.123 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.861566628999999, 2023-11-01T04:17:05.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.51741289400000001, 2023-11-01T04:17:05.123 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.124 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:05.124 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.124 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.124 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.124 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 505 2023-11-01T04:17:05.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.125 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:44.244582+0000", 2023-11-01T04:17:05.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:44.244582+0000", 2023-11-01T04:17:05.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:44.244584+0000", 2023-11-01T04:17:05.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.72e-06 2023-11-01T04:17:05.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:44.244593+0000", 2023-11-01T04:17:05.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1030000000000008e-06 2023-11-01T04:17:05.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:44.244614+0000", 2023-11-01T04:17:05.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1146e-05 2023-11-01T04:17:05.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:44.244630+0000", 2023-11-01T04:17:05.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5962999999999999e-05 2023-11-01T04:17:05.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:44.557540+0000", 2023-11-01T04:17:05.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.31291051600000003 2023-11-01T04:17:05.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:44.557832+0000", 2023-11-01T04:17:05.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00029115100000000002 2023-11-01T04:17:05.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.133 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:05.133 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:44.559114+0000", 2023-11-01T04:17:05.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012821149999999999 2023-11-01T04:17:05.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.133 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.134 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:05.134 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:44.660784+0000", 2023-11-01T04:17:05.134 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101670645 2023-11-01T04:17:05.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.134 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:05.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:44.761861+0000", 2023-11-01T04:17:05.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101076504 2023-11-01T04:17:05.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.135 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:44.761995+0000", 2023-11-01T04:17:05.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000134031 2023-11-01T04:17:05.136 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.136 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.136 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:661 3.1s0 3:8900b4ea:::smithi19039307-31:head [read 0~2801664,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2023-11-01T04:17:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:58.914334+0000", 2023-11-01T04:17:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.191814183, 2023-11-01T04:17:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.599305719, 2023-11-01T04:17:05.137 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 661 2023-11-01T04:17:05.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914334+0000", 2023-11-01T04:17:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914334+0000", 2023-11-01T04:17:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914337+0000", 2023-11-01T04:17:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9969999999999999e-06 2023-11-01T04:17:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914351+0000", 2023-11-01T04:17:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3196e-05 2023-11-01T04:17:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914384+0000", 2023-11-01T04:17:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3782000000000003e-05 2023-11-01T04:17:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914401+0000", 2023-11-01T04:17:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6983e-05 2023-11-01T04:17:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:59.014668+0000", 2023-11-01T04:17:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10026678 2023-11-01T04:17:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2023-11-01T04:17:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:59.014674+0000", 2023-11-01T04:17:05.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.57e-06 2023-11-01T04:17:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.176127+0000", 2023-11-01T04:17:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1614530809999999 2023-11-01T04:17:05.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.176262+0000", 2023-11-01T04:17:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013557900000000001 2023-11-01T04:17:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.513640+0000", 2023-11-01T04:17:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.337377751 2023-11-01T04:17:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.148 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:662 3.1s0 3:8900b4ea:::smithi19039307-31:head [read 0~2801664,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2023-11-01T04:17:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:58.914439+0000", 2023-11-01T04:17:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.1917096000000003, 2023-11-01T04:17:05.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.70047042, 2023-11-01T04:17:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.150 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 662 2023-11-01T04:17:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914439+0000", 2023-11-01T04:17:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914439+0000", 2023-11-01T04:17:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914440+0000", 2023-11-01T04:17:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1540000000000001e-06 2023-11-01T04:17:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914449+0000", 2023-11-01T04:17:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7029999999999995e-06 2023-11-01T04:17:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914468+0000", 2023-11-01T04:17:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9335999999999999e-05 2023-11-01T04:17:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914482+0000", 2023-11-01T04:17:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3723000000000001e-05 2023-11-01T04:17:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:59.014786+0000", 2023-11-01T04:17:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100304038 2023-11-01T04:17:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2023-11-01T04:17:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:59.014791+0000", 2023-11-01T04:17:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9230000000000001e-06 2023-11-01T04:17:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.280970+0000", 2023-11-01T04:17:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.266179084 2023-11-01T04:17:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.281294+0000", 2023-11-01T04:17:05.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000324147 2023-11-01T04:17:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.614909+0000", 2023-11-01T04:17:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.333615312 2023-11-01T04:17:05.160 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.161 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:663 3.1s0 3:8900b4ea:::smithi19039307-31:head [read 0~2801664,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2023-11-01T04:17:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:58.914501+0000", 2023-11-01T04:17:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.1916472410000001, 2023-11-01T04:17:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8011388340000001, 2023-11-01T04:17:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:05.162 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 663 2023-11-01T04:17:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.163 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914501+0000", 2023-11-01T04:17:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914501+0000", 2023-11-01T04:17:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914502+0000", 2023-11-01T04:17:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7300000000000004e-07 2023-11-01T04:17:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914509+0000", 2023-11-01T04:17:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3130000000000004e-06 2023-11-01T04:17:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:58.914547+0000", 2023-11-01T04:17:05.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8291999999999999e-05 2023-11-01T04:17:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:59.014678+0000", 2023-11-01T04:17:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10013066399999999 2023-11-01T04:17:05.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:59.114872+0000", 2023-11-01T04:17:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100194622 2023-11-01T04:17:05.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2023-11-01T04:17:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:59.114877+0000", 2023-11-01T04:17:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5730000000000001e-06 2023-11-01T04:17:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.282271+0000", 2023-11-01T04:17:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1673942529999999 2023-11-01T04:17:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.282510+0000", 2023-11-01T04:17:05.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023892600000000001 2023-11-01T04:17:05.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:00.715640+0000", 2023-11-01T04:17:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.43313021800000001 2023-11-01T04:17:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.173 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:733 3.1s0 3:8900b4ea:::smithi19039307-31:head [append 0~458752] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.526474+0000", 2023-11-01T04:17:05.174 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.5796743179999999, 2023-11-01T04:17:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60644699400000002, 2023-11-01T04:17:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.175 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 733 2023-11-01T04:17:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.526474+0000", 2023-11-01T04:17:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.526474+0000", 2023-11-01T04:17:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.526476+0000", 2023-11-01T04:17:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.973e-06 2023-11-01T04:17:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.527341+0000", 2023-11-01T04:17:05.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00086508699999999995 2023-11-01T04:17:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.527370+0000", 2023-11-01T04:17:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8768999999999998e-05 2023-11-01T04:17:05.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.527391+0000", 2023-11-01T04:17:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1120000000000001e-05 2023-11-01T04:17:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.532611+0000", 2023-11-01T04:17:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0052192469999999998 2023-11-01T04:17:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.532966+0000", 2023-11-01T04:17:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00035581099999999999 2023-11-01T04:17:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.535108+0000", 2023-11-01T04:17:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002141366 2023-11-01T04:17:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.642196+0000", 2023-11-01T04:17:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.107088242 2023-11-01T04:17:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.132753+0000", 2023-11-01T04:17:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.49055712600000001 2023-11-01T04:17:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.132921+0000", 2023-11-01T04:17:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016825299999999999 2023-11-01T04:17:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.187 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:734 3.1s0 3:8900b4ea:::smithi19039307-31:head [append 0~442368] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.528994+0000", 2023-11-01T04:17:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.577154699, 2023-11-01T04:17:05.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.806770294, 2023-11-01T04:17:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.189 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 734 2023-11-01T04:17:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.528994+0000", 2023-11-01T04:17:05.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.528994+0000", 2023-11-01T04:17:05.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.528996+0000", 2023-11-01T04:17:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0099999999999998e-06 2023-11-01T04:17:05.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.529729+0000", 2023-11-01T04:17:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00073339399999999995 2023-11-01T04:17:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.529754+0000", 2023-11-01T04:17:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4536e-05 2023-11-01T04:17:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.529769+0000", 2023-11-01T04:17:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5305999999999999e-05 2023-11-01T04:17:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.636750+0000", 2023-11-01T04:17:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.106980965 2023-11-01T04:17:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.636941+0000", 2023-11-01T04:17:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019043200000000001 2023-11-01T04:17:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.638493+0000", 2023-11-01T04:17:05.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001552435 2023-11-01T04:17:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.642198+0000", 2023-11-01T04:17:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003705294 2023-11-01T04:17:05.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.335430+0000", 2023-11-01T04:17:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.69323158399999996 2023-11-01T04:17:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.335764+0000", 2023-11-01T04:17:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000334338 2023-11-01T04:17:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.200 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:735 3.1s0 3:8900b4ea:::smithi19039307-31:head [append 0~262144] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.530476+0000", 2023-11-01T04:17:05.201 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.575672441, 2023-11-01T04:17:05.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.80594974699999999, 2023-11-01T04:17:05.202 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.202 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:05.202 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.202 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 735 2023-11-01T04:17:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.530476+0000", 2023-11-01T04:17:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.530476+0000", 2023-11-01T04:17:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.530478+0000", 2023-11-01T04:17:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9230000000000001e-06 2023-11-01T04:17:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.530851+0000", 2023-11-01T04:17:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00037307300000000002 2023-11-01T04:17:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.530873+0000", 2023-11-01T04:17:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1673000000000001e-05 2023-11-01T04:17:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.530885+0000", 2023-11-01T04:17:05.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2490000000000001e-05 2023-11-01T04:17:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.639708+0000", 2023-11-01T04:17:05.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.108822322 2023-11-01T04:17:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.639892+0000", 2023-11-01T04:17:05.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018433499999999999 2023-11-01T04:17:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.641072+0000", 2023-11-01T04:17:05.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011798449999999999 2023-11-01T04:17:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.032565+0000", 2023-11-01T04:17:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.39149278500000001 2023-11-01T04:17:05.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.336095+0000", 2023-11-01T04:17:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30353041400000003 2023-11-01T04:17:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.336426+0000", 2023-11-01T04:17:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033088699999999998 2023-11-01T04:17:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.214 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:736 3.1s0 3:8900b4ea:::smithi19039307-31:head [setxattr _header (57) in=7b] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.530913+0000", 2023-11-01T04:17:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.5752355570000001, 2023-11-01T04:17:05.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0088683359999999, 2023-11-01T04:17:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.216 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 736 2023-11-01T04:17:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.530913+0000", 2023-11-01T04:17:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.530913+0000", 2023-11-01T04:17:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.219 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.530914+0000", 2023-11-01T04:17:05.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.344e-06 2023-11-01T04:17:05.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.530936+0000", 2023-11-01T04:17:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1492e-05 2023-11-01T04:17:05.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.530955+0000", 2023-11-01T04:17:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8940000000000002e-05 2023-11-01T04:17:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.222 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.222 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.530970+0000", 2023-11-01T04:17:05.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5036e-05 2023-11-01T04:17:05.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.222 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.742333+0000", 2023-11-01T04:17:05.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.211363466 2023-11-01T04:17:05.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.742544+0000", 2023-11-01T04:17:05.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000210218 2023-11-01T04:17:05.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:05.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.743401+0000", 2023-11-01T04:17:05.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00085738100000000005 2023-11-01T04:17:05.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:05.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.133027+0000", 2023-11-01T04:17:05.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.389626368 2023-11-01T04:17:05.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.226 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.226 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:05.226 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.539414+0000", 2023-11-01T04:17:05.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40638684800000002 2023-11-01T04:17:05.227 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.227 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.539781+0000", 2023-11-01T04:17:05.228 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00036724300000000001 2023-11-01T04:17:05.228 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.228 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.228 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.229 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:737 3.1s0 3:8900b4ea:::smithi19039307-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:05.229 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.530993+0000", 2023-11-01T04:17:05.229 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.5751561619999999, 2023-11-01T04:17:05.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.112781086, 2023-11-01T04:17:05.230 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.230 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:05.230 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.230 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.230 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.231 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 737 2023-11-01T04:17:05.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.231 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.530993+0000", 2023-11-01T04:17:05.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.232 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.530993+0000", 2023-11-01T04:17:05.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.233 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.530994+0000", 2023-11-01T04:17:05.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.097e-06 2023-11-01T04:17:05.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.234 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.234 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.530999+0000", 2023-11-01T04:17:05.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7830000000000004e-06 2023-11-01T04:17:05.234 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.235 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.235 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531027+0000", 2023-11-01T04:17:05.235 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8109000000000001e-05 2023-11-01T04:17:05.235 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.236 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.236 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531054+0000", 2023-11-01T04:17:05.236 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6761999999999999e-05 2023-11-01T04:17:05.236 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.236 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.236 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.744087+0000", 2023-11-01T04:17:05.237 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.21303244499999999 2023-11-01T04:17:05.237 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.237 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:05.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.744280+0000", 2023-11-01T04:17:05.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019295200000000001 2023-11-01T04:17:05.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.540784+0000", 2023-11-01T04:17:05.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.79650395900000004 2023-11-01T04:17:05.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.239 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.239 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.541115+0000", 2023-11-01T04:17:05.239 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033175400000000002 2023-11-01T04:17:05.240 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.240 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.240 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.643774+0000", 2023-11-01T04:17:05.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10265822500000001 2023-11-01T04:17:05.240 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.241 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.241 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.241 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:738 3.1s0 3:8bb0e774:::smithi19039307-12:28 [read 0~196608,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:05.242 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.531082+0000", 2023-11-01T04:17:05.242 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.575066874, 2023-11-01T04:17:05.242 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.90639658599999995, 2023-11-01T04:17:05.242 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.242 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:05.242 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.243 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.243 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.243 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 738 2023-11-01T04:17:05.243 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.243 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.244 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.244 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531082+0000", 2023-11-01T04:17:05.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.244 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.244 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.245 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531082+0000", 2023-11-01T04:17:05.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.245 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.245 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.245 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531083+0000", 2023-11-01T04:17:05.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.223e-06 2023-11-01T04:17:05.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531095+0000", 2023-11-01T04:17:05.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2223e-05 2023-11-01T04:17:05.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.247 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531115+0000", 2023-11-01T04:17:05.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0126e-05 2023-11-01T04:17:05.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.248 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.248 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531141+0000", 2023-11-01T04:17:05.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5452999999999998e-05 2023-11-01T04:17:05.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.249 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.249 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.745322+0000", 2023-11-01T04:17:05.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.21418141700000001 2023-11-01T04:17:05.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.745617+0000", 2023-11-01T04:17:05.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00029428899999999999 2023-11-01T04:17:05.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.437478+0000", 2023-11-01T04:17:05.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.69186185499999997 2023-11-01T04:17:05.252 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.252 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.252 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.252 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:739 3.1s0 3:8bb0e774:::smithi19039307-12:28 [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:05.253 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.531168+0000", 2023-11-01T04:17:05.253 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.574980517, 2023-11-01T04:17:05.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.90701792199999998, 2023-11-01T04:17:05.253 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.253 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:05.254 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.254 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.254 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.254 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 739 2023-11-01T04:17:05.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.255 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.255 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.255 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.255 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531168+0000", 2023-11-01T04:17:05.255 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531168+0000", 2023-11-01T04:17:05.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.257 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.257 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.257 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531169+0000", 2023-11-01T04:17:05.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1429999999999999e-06 2023-11-01T04:17:05.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.258 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531176+0000", 2023-11-01T04:17:05.258 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8770000000000004e-06 2023-11-01T04:17:05.258 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531201+0000", 2023-11-01T04:17:05.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5103e-05 2023-11-01T04:17:05.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.260 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531216+0000", 2023-11-01T04:17:05.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5082999999999999e-05 2023-11-01T04:17:05.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.261 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.261 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.747639+0000", 2023-11-01T04:17:05.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.216422537 2023-11-01T04:17:05.261 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.261 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.262 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.747815+0000", 2023-11-01T04:17:05.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017635200000000001 2023-11-01T04:17:05.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.438186+0000", 2023-11-01T04:17:05.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.69037082699999996 2023-11-01T04:17:05.263 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.263 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.263 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.264 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.264 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:740 3.1s0 3:8bb0e774:::smithi19039307-12:28 [read 0~196608,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:05.264 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.531241+0000", 2023-11-01T04:17:05.264 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.5749078189999999, 2023-11-01T04:17:05.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.007880933, 2023-11-01T04:17:05.265 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.265 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:05.265 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.265 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.265 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.265 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 740 2023-11-01T04:17:05.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.266 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531241+0000", 2023-11-01T04:17:05.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.267 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.267 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531241+0000", 2023-11-01T04:17:05.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.268 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.268 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531242+0000", 2023-11-01T04:17:05.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1000000000000001e-06 2023-11-01T04:17:05.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531248+0000", 2023-11-01T04:17:05.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4400000000000002e-06 2023-11-01T04:17:05.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.270 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531268+0000", 2023-11-01T04:17:05.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9157e-05 2023-11-01T04:17:05.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.531288+0000", 2023-11-01T04:17:05.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0341999999999999e-05 2023-11-01T04:17:05.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.272 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.272 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.748872+0000", 2023-11-01T04:17:05.272 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.21758458999999999 2023-11-01T04:17:05.272 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.273 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.273 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.749041+0000", 2023-11-01T04:17:05.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016822500000000001 2023-11-01T04:17:05.273 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.273 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.273 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.274 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.539122+0000", 2023-11-01T04:17:05.274 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.79008107900000002 2023-11-01T04:17:05.274 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.274 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.274 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.274 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.275 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.275 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.1s0 82/78 ECSubReadReply(tid=116, attrs_read=0))", 2023-11-01T04:17:05.275 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.849765+0000", 2023-11-01T04:17:05.275 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.2563837659999999, 2023-11-01T04:17:05.275 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.48415446200000001, 2023-11-01T04:17:05.276 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.276 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:05.276 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.276 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.276 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.849765+0000", 2023-11-01T04:17:05.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.277 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.277 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.277 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.849765+0000", 2023-11-01T04:17:05.278 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.278 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.278 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.278 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.278 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.849766+0000", 2023-11-01T04:17:05.279 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.64e-07 2023-11-01T04:17:05.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.279 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.279 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.849826+0000", 2023-11-01T04:17:05.280 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0087999999999998e-05 2023-11-01T04:17:05.280 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.280 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.280 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.280 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.849841+0000", 2023-11-01T04:17:05.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5546e-05 2023-11-01T04:17:05.281 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.281 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.281 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.281 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.849850+0000", 2023-11-01T04:17:05.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2039999999999997e-06 2023-11-01T04:17:05.282 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.282 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.282 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.282 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.333821+0000", 2023-11-01T04:17:05.282 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.48397107499999997 2023-11-01T04:17:05.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.283 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.283 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.283 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.333919+0000", 2023-11-01T04:17:05.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8585000000000006e-05 2023-11-01T04:17:05.283 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.284 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.284 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.284 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.284 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.284 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 82/78 ECSubWriteReply(tid=113, last_complete=82'67, committed=1, applied=1))", 2023-11-01T04:17:05.285 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.849864+0000", 2023-11-01T04:17:05.285 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.2562842320000001, 2023-11-01T04:17:05.285 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.48418772399999999, 2023-11-01T04:17:05.285 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.285 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:05.285 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.286 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.286 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.849864+0000", 2023-11-01T04:17:05.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.286 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.849864+0000", 2023-11-01T04:17:05.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.288 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.288 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.849865+0000", 2023-11-01T04:17:05.288 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.44e-07 2023-11-01T04:17:05.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.288 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.289 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.849869+0000", 2023-11-01T04:17:05.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0430000000000001e-06 2023-11-01T04:17:05.289 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.289 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.289 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.289 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.849882+0000", 2023-11-01T04:17:05.290 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2696e-05 2023-11-01T04:17:05.290 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.290 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.849889+0000", 2023-11-01T04:17:05.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3540000000000002e-06 2023-11-01T04:17:05.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.291 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.334009+0000", 2023-11-01T04:17:05.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.484120255 2023-11-01T04:17:05.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.334052+0000", 2023-11-01T04:17:05.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2831999999999999e-05 2023-11-01T04:17:05.293 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.293 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.293 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.293 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.1s0 82/78 ECSubReadReply(tid=114, attrs_read=0))", 2023-11-01T04:17:05.294 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.938358+0000", 2023-11-01T04:17:05.294 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.167790383, 2023-11-01T04:17:05.294 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.499148127, 2023-11-01T04:17:05.294 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.294 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:05.295 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.295 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938358+0000", 2023-11-01T04:17:05.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.295 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938358+0000", 2023-11-01T04:17:05.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.297 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.297 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938359+0000", 2023-11-01T04:17:05.297 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6700000000000002e-07 2023-11-01T04:17:05.297 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.298 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.298 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938635+0000", 2023-11-01T04:17:05.298 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027598800000000001 2023-11-01T04:17:05.298 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.298 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.298 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.299 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938650+0000", 2023-11-01T04:17:05.299 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5258e-05 2023-11-01T04:17:05.299 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.299 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.299 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.300 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938660+0000", 2023-11-01T04:17:05.300 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0088999999999999e-05 2023-11-01T04:17:05.300 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.300 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.300 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.300 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.436674+0000", 2023-11-01T04:17:05.301 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.49801308799999999 2023-11-01T04:17:05.301 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.301 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.301 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.301 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.437506+0000", 2023-11-01T04:17:05.302 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00083283700000000001 2023-11-01T04:17:05.302 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.302 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.302 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.302 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.303 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.303 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.1s0 82/78 ECSubReadReply(tid=115, attrs_read=0))", 2023-11-01T04:17:05.303 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.938678+0000", 2023-11-01T04:17:05.303 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.167470376, 2023-11-01T04:17:05.303 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.49952782400000001, 2023-11-01T04:17:05.303 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.304 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:05.304 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.304 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.304 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.304 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938678+0000", 2023-11-01T04:17:05.304 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.305 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.305 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.305 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.305 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938678+0000", 2023-11-01T04:17:05.305 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.305 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.306 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.306 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.306 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938679+0000", 2023-11-01T04:17:05.306 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9500000000000003e-07 2023-11-01T04:17:05.306 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.307 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.307 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.307 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938712+0000", 2023-11-01T04:17:05.307 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3565999999999997e-05 2023-11-01T04:17:05.307 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.307 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.308 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.308 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938723+0000", 2023-11-01T04:17:05.308 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0757000000000001e-05 2023-11-01T04:17:05.308 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.308 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.308 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.309 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938729+0000", 2023-11-01T04:17:05.309 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3640000000000004e-06 2023-11-01T04:17:05.309 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.309 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.309 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.437555+0000", 2023-11-01T04:17:05.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.49882554699999998 2023-11-01T04:17:05.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.438206+0000", 2023-11-01T04:17:05.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00065109499999999997 2023-11-01T04:17:05.311 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.311 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.311 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.312 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.312 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.1s0 82/78 ECSubReadReply(tid=116, attrs_read=0))", 2023-11-01T04:17:05.312 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.938778+0000", 2023-11-01T04:17:05.312 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.1673704069999999, 2023-11-01T04:17:05.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60036954099999995, 2023-11-01T04:17:05.312 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.313 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:05.313 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.313 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.313 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.313 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938778+0000", 2023-11-01T04:17:05.313 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.314 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.314 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.314 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.314 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938778+0000", 2023-11-01T04:17:05.314 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.315 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.315 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938779+0000", 2023-11-01T04:17:05.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2500000000000005e-07 2023-11-01T04:17:05.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.316 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.316 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938842+0000", 2023-11-01T04:17:05.316 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3213999999999995e-05 2023-11-01T04:17:05.316 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.317 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.317 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938855+0000", 2023-11-01T04:17:05.317 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3336000000000001e-05 2023-11-01T04:17:05.317 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.317 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.318 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.318 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938863+0000", 2023-11-01T04:17:05.318 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5229999999999998e-06 2023-11-01T04:17:05.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.318 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.538411+0000", 2023-11-01T04:17:05.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.59954772999999995 2023-11-01T04:17:05.319 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.319 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.319 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.539148+0000", 2023-11-01T04:17:05.320 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00073711299999999996 2023-11-01T04:17:05.320 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.320 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.320 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.321 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.321 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.321 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 82/78 ECSubWriteReply(tid=113, last_complete=82'67, committed=1, applied=1))", 2023-11-01T04:17:05.321 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.938882+0000", 2023-11-01T04:17:05.321 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.167266175, 2023-11-01T04:17:05.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60035851900000003, 2023-11-01T04:17:05.322 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.322 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:05.322 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.322 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.323 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.323 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938882+0000", 2023-11-01T04:17:05.323 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.323 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.323 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.323 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.324 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938882+0000", 2023-11-01T04:17:05.324 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.324 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.325 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938883+0000", 2023-11-01T04:17:05.325 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0999999999999999e-07 2023-11-01T04:17:05.325 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.325 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.325 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938886+0000", 2023-11-01T04:17:05.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.517e-06 2023-11-01T04:17:05.326 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.326 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.326 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.327 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938897+0000", 2023-11-01T04:17:05.327 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0395e-05 2023-11-01T04:17:05.327 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.327 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.938902+0000", 2023-11-01T04:17:05.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4550000000000003e-06 2023-11-01T04:17:05.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.539202+0000", 2023-11-01T04:17:05.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60029959300000002 2023-11-01T04:17:05.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.539241+0000", 2023-11-01T04:17:05.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9149000000000001e-05 2023-11-01T04:17:05.330 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.330 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.330 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.331 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 83/78 ECSubWriteReply(tid=113, last_complete=82'67, committed=1, applied=1))", 2023-11-01T04:17:05.331 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.939638+0000", 2023-11-01T04:17:05.331 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.1665106860000001, 2023-11-01T04:17:05.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60093730000000001, 2023-11-01T04:17:05.331 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.332 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:05.332 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.939638+0000", 2023-11-01T04:17:05.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.939638+0000", 2023-11-01T04:17:05.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.939638+0000", 2023-11-01T04:17:05.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3000000000000001e-07 2023-11-01T04:17:05.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.939642+0000", 2023-11-01T04:17:05.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1269999999999999e-06 2023-11-01T04:17:05.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.939652+0000", 2023-11-01T04:17:05.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.044e-05 2023-11-01T04:17:05.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.939658+0000", 2023-11-01T04:17:05.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.319e-06 2023-11-01T04:17:05.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.539285+0000", 2023-11-01T04:17:05.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.59962644300000001 2023-11-01T04:17:05.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.540575+0000", 2023-11-01T04:17:05.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012905410000000001 2023-11-01T04:17:05.339 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.339 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.339 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.339 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.339 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.340 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:17:05.340 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:05.341 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-01T04:17:05.442 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:05.440+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:05.517 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-01T04:17:05.518 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:17:05.519 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2023-11-01T04:17:05.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.519 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 84/31 ECSubWriteReply(tid=150, last_complete=84'63, committed=1, applied=1))", 2023-11-01T04:17:05.519 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:05.479510+0000", 2023-11-01T04:17:05.519 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.039394228000000003, 2023-11-01T04:17:05.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.039413124000000001, 2023-11-01T04:17:05.520 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.520 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2023-11-01T04:17:05.520 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:05.479510+0000", 2023-11-01T04:17:05.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.521 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.522 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:05.479510+0000", 2023-11-01T04:17:05.522 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.522 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.522 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:05.479511+0000", 2023-11-01T04:17:05.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0999999999999997e-07 2023-11-01T04:17:05.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:05.479517+0000", 2023-11-01T04:17:05.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9170000000000003e-06 2023-11-01T04:17:05.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:05.479538+0000", 2023-11-01T04:17:05.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0336e-05 2023-11-01T04:17:05.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.526 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:05.479555+0000", 2023-11-01T04:17:05.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7339000000000001e-05 2023-11-01T04:17:05.526 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.526 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.526 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.527 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 84/31 ECSubWriteReply(tid=150, last_complete=84'63, committed=1, applied=1))", 2023-11-01T04:17:05.527 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:05.479653+0000", 2023-11-01T04:17:05.527 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.039251645000000002, 2023-11-01T04:17:05.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.039299908000000001, 2023-11-01T04:17:05.528 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.528 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2023-11-01T04:17:05.528 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.528 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.528 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:05.479653+0000", 2023-11-01T04:17:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.529 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:05.479653+0000", 2023-11-01T04:17:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:05.479654+0000", 2023-11-01T04:17:05.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0499999999999999e-06 2023-11-01T04:17:05.531 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.531 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.531 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:05.479661+0000", 2023-11-01T04:17:05.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5869999999999998e-06 2023-11-01T04:17:05.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:05.479681+0000", 2023-11-01T04:17:05.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0313e-05 2023-11-01T04:17:05.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:05.479696+0000", 2023-11-01T04:17:05.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4599e-05 2023-11-01T04:17:05.533 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.534 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.534 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.534 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.534 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2023-11-01T04:17:05.534 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 2 2023-11-01T04:17:05.534 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:17:05.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-01T04:17:05.543 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:05.540+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:05.644 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:17:05.643+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:17:05.652 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:17:05.652 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:17:05.652 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2023-11-01T04:17:05.652 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2023-11-01T04:17:05.652 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:17:05.666 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-01T04:17:05.689 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:17:05.689 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-11-01T04:17:05.709 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:17:05.710 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-01T04:17:05.729 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:17:05.729 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-01T04:17:05.744 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:17:05.742+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:17:05.750 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:17:05.750 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-01T04:17:05.768 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:17:05.768 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-01T04:17:05.790 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:17:05.790 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-01T04:17:05.793 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:17:05.793 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2023-11-01T04:17:05.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2023-11-01T04:17:05.794 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2023-11-01T04:17:05.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.794 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:198 3.4s0 3:2f6e6eee:::smithi19039307-7:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:05.794 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.965004+0000", 2023-11-01T04:17:05.794 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 48.824969555000003, 2023-11-01T04:17:05.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.37915321499999999, 2023-11-01T04:17:05.795 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.795 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:05.795 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.795 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.796 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.796 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 198 2023-11-01T04:17:05.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.796 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965004+0000", 2023-11-01T04:17:05.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965004+0000", 2023-11-01T04:17:05.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965005+0000", 2023-11-01T04:17:05.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8599999999999997e-07 2023-11-01T04:17:05.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965011+0000", 2023-11-01T04:17:05.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1970000000000001e-06 2023-11-01T04:17:05.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965117+0000", 2023-11-01T04:17:05.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000105919 2023-11-01T04:17:05.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965150+0000", 2023-11-01T04:17:05.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2468999999999997e-05 2023-11-01T04:17:05.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.041140+0000", 2023-11-01T04:17:05.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.075989950000000001 2023-11-01T04:17:05.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.041378+0000", 2023-11-01T04:17:05.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023841499999999999 2023-11-01T04:17:05.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:05.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.042565+0000", 2023-11-01T04:17:05.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011867480000000001 2023-11-01T04:17:05.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:05.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.145152+0000", 2023-11-01T04:17:05.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102587356 2023-11-01T04:17:05.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:05.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.344048+0000", 2023-11-01T04:17:05.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19889615599999999 2023-11-01T04:17:05.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.344158+0000", 2023-11-01T04:17:05.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000109219 2023-11-01T04:17:05.807 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.807 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.807 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.808 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:369 3.4s0 3:22b97aa6:::smithi19039307-42:head [append 0~196608] snapc d=[d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:17:05.808 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.064665+0000", 2023-11-01T04:17:05.808 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 37.725309318000001, 2023-11-01T04:17:05.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40249887899999998, 2023-11-01T04:17:05.809 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.809 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:05.809 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.809 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.810 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.810 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 369 2023-11-01T04:17:05.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.810 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.064665+0000", 2023-11-01T04:17:05.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.064665+0000", 2023-11-01T04:17:05.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.064666+0000", 2023-11-01T04:17:05.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.64e-06 2023-11-01T04:17:05.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.065011+0000", 2023-11-01T04:17:05.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034451099999999998 2023-11-01T04:17:05.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.065036+0000", 2023-11-01T04:17:05.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5148999999999999e-05 2023-11-01T04:17:05.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165152+0000", 2023-11-01T04:17:05.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100115738 2023-11-01T04:17:05.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.167023+0000", 2023-11-01T04:17:05.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001871075 2023-11-01T04:17:05.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.167186+0000", 2023-11-01T04:17:05.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016346199999999999 2023-11-01T04:17:05.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:05.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.168223+0000", 2023-11-01T04:17:05.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010366119999999999 2023-11-01T04:17:05.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:05.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.270989+0000", 2023-11-01T04:17:05.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102765841 2023-11-01T04:17:05.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:05.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.466957+0000", 2023-11-01T04:17:05.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.195968693 2023-11-01T04:17:05.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.467163+0000", 2023-11-01T04:17:05.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020615799999999999 2023-11-01T04:17:05.821 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.822 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.822 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.822 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:370 3.4s0 3:22b97aa6:::smithi19039307-42:head [setxattr _header (56) in=7b] snapc d=[d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:17:05.823 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.165200+0000", 2023-11-01T04:17:05.823 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 37.624774252000002, 2023-11-01T04:17:05.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40210787999999997, 2023-11-01T04:17:05.823 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.824 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:05.824 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.824 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.824 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.824 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 370 2023-11-01T04:17:05.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.825 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165200+0000", 2023-11-01T04:17:05.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165200+0000", 2023-11-01T04:17:05.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165202+0000", 2023-11-01T04:17:05.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6000000000000001e-06 2023-11-01T04:17:05.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165215+0000", 2023-11-01T04:17:05.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2904e-05 2023-11-01T04:17:05.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165249+0000", 2023-11-01T04:17:05.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3655000000000002e-05 2023-11-01T04:17:05.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165268+0000", 2023-11-01T04:17:05.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8810000000000001e-05 2023-11-01T04:17:05.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.169297+0000", 2023-11-01T04:17:05.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040292280000000001 2023-11-01T04:17:05.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.169474+0000", 2023-11-01T04:17:05.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017749500000000001 2023-11-01T04:17:05.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:05.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.170331+0000", 2023-11-01T04:17:05.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00085666300000000002 2023-11-01T04:17:05.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:05.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.271064+0000", 2023-11-01T04:17:05.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10073283099999999 2023-11-01T04:17:05.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:05.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.567166+0000", 2023-11-01T04:17:05.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29610207100000002 2023-11-01T04:17:05.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.567307+0000", 2023-11-01T04:17:05.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014162300000000001 2023-11-01T04:17:05.836 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.836 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.836 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.837 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:371 3.4s0 3:22b97aa6:::smithi19039307-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:17:05.837 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.165296+0000", 2023-11-01T04:17:05.837 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 37.624678174000003, 2023-11-01T04:17:05.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60529993999999998, 2023-11-01T04:17:05.837 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.838 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:05.838 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.838 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.838 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.839 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 371 2023-11-01T04:17:05.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.839 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165296+0000", 2023-11-01T04:17:05.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165296+0000", 2023-11-01T04:17:05.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165297+0000", 2023-11-01T04:17:05.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4729999999999999e-06 2023-11-01T04:17:05.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165304+0000", 2023-11-01T04:17:05.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7730000000000002e-06 2023-11-01T04:17:05.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165338+0000", 2023-11-01T04:17:05.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.392e-05 2023-11-01T04:17:05.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165353+0000", 2023-11-01T04:17:05.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4922e-05 2023-11-01T04:17:05.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.266752+0000", 2023-11-01T04:17:05.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101399533 2023-11-01T04:17:05.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:05.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.266921+0000", 2023-11-01T04:17:05.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016895199999999999 2023-11-01T04:17:05.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.568170+0000", 2023-11-01T04:17:05.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30124910799999999 2023-11-01T04:17:05.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.568436+0000", 2023-11-01T04:17:05.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00026584599999999998 2023-11-01T04:17:05.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.770596+0000", 2023-11-01T04:17:05.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20215941300000001 2023-11-01T04:17:05.849 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.849 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.849 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.850 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:404 3.4s0 3:22b97aa6:::smithi19039307-42:head [read 0~1900544,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2023-11-01T04:17:05.850 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:33.200147+0000", 2023-11-01T04:17:05.850 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 32.589826690000002, 2023-11-01T04:17:05.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40700329400000002, 2023-11-01T04:17:05.850 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.851 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:05.851 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.851 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.851 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.851 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 404 2023-11-01T04:17:05.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.852 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200147+0000", 2023-11-01T04:17:05.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200147+0000", 2023-11-01T04:17:05.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200149+0000", 2023-11-01T04:17:05.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.187e-06 2023-11-01T04:17:05.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200162+0000", 2023-11-01T04:17:05.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2943e-05 2023-11-01T04:17:05.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200189+0000", 2023-11-01T04:17:05.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7056e-05 2023-11-01T04:17:05.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200209+0000", 2023-11-01T04:17:05.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9802000000000001e-05 2023-11-01T04:17:05.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.403941+0000", 2023-11-01T04:17:05.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20373233099999999 2023-11-01T04:17:05.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.404113+0000", 2023-11-01T04:17:05.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000171855 2023-11-01T04:17:05.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.607150+0000", 2023-11-01T04:17:05.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20303711999999999 2023-11-01T04:17:05.860 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.860 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.860 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.861 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:449 3.4s0 3:2f6e6eee:::smithi19039307-7:15 [read 0~114688,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2023-11-01T04:17:05.861 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:39.142570+0000", 2023-11-01T04:17:05.861 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.647403778000001, 2023-11-01T04:17:05.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41050975099999998, 2023-11-01T04:17:05.862 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.862 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:05.862 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.862 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.863 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.863 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 449 2023-11-01T04:17:05.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.863 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.142570+0000", 2023-11-01T04:17:05.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.142570+0000", 2023-11-01T04:17:05.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.142571+0000", 2023-11-01T04:17:05.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.097e-06 2023-11-01T04:17:05.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.142578+0000", 2023-11-01T04:17:05.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.046e-06 2023-11-01T04:17:05.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.142600+0000", 2023-11-01T04:17:05.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1676999999999999e-05 2023-11-01T04:17:05.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.142614+0000", 2023-11-01T04:17:05.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4479e-05 2023-11-01T04:17:05.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.142867+0000", 2023-11-01T04:17:05.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025300499999999999 2023-11-01T04:17:05.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:05.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.142885+0000", 2023-11-01T04:17:05.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7385999999999999e-05 2023-11-01T04:17:05.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.450338+0000", 2023-11-01T04:17:05.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.307453219 2023-11-01T04:17:05.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.450500+0000", 2023-11-01T04:17:05.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016191 2023-11-01T04:17:05.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.553080+0000", 2023-11-01T04:17:05.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102579932 2023-11-01T04:17:05.873 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.873 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.873 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.874 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:547 3.4s0 3:2c54b66d:::smithi19039307-30:head [append 0~753664] snapc 1e=[1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:05.874 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:47.808599+0000", 2023-11-01T04:17:05.874 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 17.981374924000001, 2023-11-01T04:17:05.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41812574499999999, 2023-11-01T04:17:05.875 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.875 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:05.875 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.875 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.875 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.876 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 547 2023-11-01T04:17:05.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.876 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.808599+0000", 2023-11-01T04:17:05.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.808599+0000", 2023-11-01T04:17:05.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.808602+0000", 2023-11-01T04:17:05.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9859999999999999e-06 2023-11-01T04:17:05.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.809960+0000", 2023-11-01T04:17:05.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013583950000000001 2023-11-01T04:17:05.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.809995+0000", 2023-11-01T04:17:05.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4452000000000002e-05 2023-11-01T04:17:05.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.910089+0000", 2023-11-01T04:17:05.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100094554 2023-11-01T04:17:05.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.910257+0000", 2023-11-01T04:17:05.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016809600000000001 2023-11-01T04:17:05.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.910547+0000", 2023-11-01T04:17:05.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000289552 2023-11-01T04:17:05.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:05.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.913558+0000", 2023-11-01T04:17:05.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030110029999999999 2023-11-01T04:17:05.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:05.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.922892+0000", 2023-11-01T04:17:05.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0093338620000000001 2023-11-01T04:17:05.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:05.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.226596+0000", 2023-11-01T04:17:05.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30370408399999999 2023-11-01T04:17:05.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.226725+0000", 2023-11-01T04:17:05.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000128761 2023-11-01T04:17:05.887 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.887 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.888 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.888 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:548 3.4s0 3:2c54b66d:::smithi19039307-30:head [append 0~720896] snapc 1e=[1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:05.888 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:47.910162+0000", 2023-11-01T04:17:05.889 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 17.879811699000001, 2023-11-01T04:17:05.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41704092300000001, 2023-11-01T04:17:05.889 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.889 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:05.890 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.890 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.890 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.890 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 548 2023-11-01T04:17:05.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.891 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.910162+0000", 2023-11-01T04:17:05.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.910162+0000", 2023-11-01T04:17:05.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.910165+0000", 2023-11-01T04:17:05.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8930000000000001e-06 2023-11-01T04:17:05.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911468+0000", 2023-11-01T04:17:05.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013033599999999999 2023-11-01T04:17:05.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911502+0000", 2023-11-01T04:17:05.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3108999999999998e-05 2023-11-01T04:17:05.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911523+0000", 2023-11-01T04:17:05.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1282999999999999e-05 2023-11-01T04:17:05.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.915890+0000", 2023-11-01T04:17:05.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004367541 2023-11-01T04:17:05.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.916205+0000", 2023-11-01T04:17:05.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00031428400000000002 2023-11-01T04:17:05.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:05.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.918707+0000", 2023-11-01T04:17:05.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025018929999999998 2023-11-01T04:17:05.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:05.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.025371+0000", 2023-11-01T04:17:05.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.106664498 2023-11-01T04:17:05.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:05.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.327079+0000", 2023-11-01T04:17:05.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30170833499999999 2023-11-01T04:17:05.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.327203+0000", 2023-11-01T04:17:05.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012372699999999999 2023-11-01T04:17:05.901 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.901 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.902 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.902 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:549 3.4s0 3:2c54b66d:::smithi19039307-30:head [append 0~49152] snapc 1e=[1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:05.902 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:47.911560+0000", 2023-11-01T04:17:05.903 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 17.878413861999999, 2023-11-01T04:17:05.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.415940221, 2023-11-01T04:17:05.903 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.903 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:05.903 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.903 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.904 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.904 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 549 2023-11-01T04:17:05.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.904 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911560+0000", 2023-11-01T04:17:05.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911560+0000", 2023-11-01T04:17:05.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911562+0000", 2023-11-01T04:17:05.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0600000000000002e-06 2023-11-01T04:17:05.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911676+0000", 2023-11-01T04:17:05.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114405 2023-11-01T04:17:05.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911698+0000", 2023-11-01T04:17:05.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1625999999999999e-05 2023-11-01T04:17:05.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911718+0000", 2023-11-01T04:17:05.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9706000000000001e-05 2023-11-01T04:17:05.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.920665+0000", 2023-11-01T04:17:05.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0089476859999999998 2023-11-01T04:17:05.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.920862+0000", 2023-11-01T04:17:05.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000196443 2023-11-01T04:17:05.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:05.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.921831+0000", 2023-11-01T04:17:05.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00096887500000000001 2023-11-01T04:17:05.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:05.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.025374+0000", 2023-11-01T04:17:05.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103543049 2023-11-01T04:17:05.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:05.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.327401+0000", 2023-11-01T04:17:05.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30202717899999998 2023-11-01T04:17:05.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.327500+0000", 2023-11-01T04:17:05.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9191999999999995e-05 2023-11-01T04:17:05.916 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.916 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.916 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.917 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:550 3.4s0 3:2c54b66d:::smithi19039307-30:head [setxattr _header (57) in=7b] snapc 1e=[1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:05.917 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:47.911747+0000", 2023-11-01T04:17:05.917 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 17.878226715, 2023-11-01T04:17:05.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41617588700000002, 2023-11-01T04:17:05.917 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.918 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:05.918 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.918 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.918 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.918 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 550 2023-11-01T04:17:05.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.919 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911747+0000", 2023-11-01T04:17:05.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911747+0000", 2023-11-01T04:17:05.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911749+0000", 2023-11-01T04:17:05.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7e-06 2023-11-01T04:17:05.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911756+0000", 2023-11-01T04:17:05.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5490000000000001e-06 2023-11-01T04:17:05.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911776+0000", 2023-11-01T04:17:05.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9150000000000001e-05 2023-11-01T04:17:05.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911791+0000", 2023-11-01T04:17:05.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5407e-05 2023-11-01T04:17:05.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.923195+0000", 2023-11-01T04:17:05.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011403583 2023-11-01T04:17:05.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.923372+0000", 2023-11-01T04:17:05.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000177207 2023-11-01T04:17:05.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.927 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:05.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.924202+0000", 2023-11-01T04:17:05.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00083057400000000005 2023-11-01T04:17:05.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.927 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:05.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.025376+0000", 2023-11-01T04:17:05.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101173993 2023-11-01T04:17:05.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:05.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.327778+0000", 2023-11-01T04:17:05.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30240209600000001 2023-11-01T04:17:05.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.327923+0000", 2023-11-01T04:17:05.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000144628 2023-11-01T04:17:05.930 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.930 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.930 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.931 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:551 3.4s0 3:2c54b66d:::smithi19039307-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:05.931 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:47.911849+0000", 2023-11-01T04:17:05.931 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 17.878124351, 2023-11-01T04:17:05.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.72034387799999999, 2023-11-01T04:17:05.932 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.932 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:05.932 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.932 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.932 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.932 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 551 2023-11-01T04:17:05.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.933 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.933 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.933 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911849+0000", 2023-11-01T04:17:05.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.934 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.934 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911849+0000", 2023-11-01T04:17:05.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.935 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.935 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.935 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911851+0000", 2023-11-01T04:17:05.935 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.477e-06 2023-11-01T04:17:05.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.936 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911858+0000", 2023-11-01T04:17:05.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.063e-06 2023-11-01T04:17:05.936 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.937 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.937 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.937 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911900+0000", 2023-11-01T04:17:05.937 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1505999999999999e-05 2023-11-01T04:17:05.937 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.937 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.938 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.938 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911919+0000", 2023-11-01T04:17:05.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9550000000000001e-05 2023-11-01T04:17:05.938 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.024812+0000", 2023-11-01T04:17:05.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.112892837 2023-11-01T04:17:05.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:05.940 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.024988+0000", 2023-11-01T04:17:05.940 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017627699999999999 2023-11-01T04:17:05.940 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.940 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.940 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.941 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.328738+0000", 2023-11-01T04:17:05.941 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30374931799999999 2023-11-01T04:17:05.941 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.941 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.941 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.942 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.328956+0000", 2023-11-01T04:17:05.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021849900000000001 2023-11-01T04:17:05.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.942 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.942 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.942 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.632193+0000", 2023-11-01T04:17:05.943 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30323735099999999 2023-11-01T04:17:05.943 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.943 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.943 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.943 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.944 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:624 3.bs0 3:d1d3cdc7:::smithi19039307-26:head [append 0~0] snapc 27=[27,26,22,1b,19,18,12,11] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2023-11-01T04:17:05.944 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:54.789464+0000", 2023-11-01T04:17:05.944 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.000510345, 2023-11-01T04:17:05.944 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41123595400000001, 2023-11-01T04:17:05.944 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.945 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:05.945 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.945 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.945 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.945 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 624 2023-11-01T04:17:05.945 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.946 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.946 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.946 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.946 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789464+0000", 2023-11-01T04:17:05.946 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.946 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.947 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.947 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.947 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789464+0000", 2023-11-01T04:17:05.947 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.947 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.948 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.948 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.948 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789466+0000", 2023-11-01T04:17:05.948 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9399999999999998e-06 2023-11-01T04:17:05.948 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.948 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.949 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.949 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789479+0000", 2023-11-01T04:17:05.949 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2595999999999999e-05 2023-11-01T04:17:05.949 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.949 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.949 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.950 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789514+0000", 2023-11-01T04:17:05.950 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4845999999999998e-05 2023-11-01T04:17:05.950 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.950 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.950 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.951 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789531+0000", 2023-11-01T04:17:05.951 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6799999999999998e-05 2023-11-01T04:17:05.951 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.951 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.951 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.952 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789669+0000", 2023-11-01T04:17:05.952 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013866099999999999 2023-11-01T04:17:05.952 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.952 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.952 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2023-11-01T04:17:05.952 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789686+0000", 2023-11-01T04:17:05.953 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7056999999999999e-05 2023-11-01T04:17:05.953 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.953 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.953 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.953 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.891232+0000", 2023-11-01T04:17:05.954 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101545627 2023-11-01T04:17:05.954 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.954 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.954 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.954 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.891421+0000", 2023-11-01T04:17:05.954 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018915100000000001 2023-11-01T04:17:05.955 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.955 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.955 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:05.955 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.893726+0000", 2023-11-01T04:17:05.955 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023044739999999999 2023-11-01T04:17:05.955 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.956 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.956 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:05.956 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.999788+0000", 2023-11-01T04:17:05.956 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.106062225 2023-11-01T04:17:05.956 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.956 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.957 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:05.957 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.200471+0000", 2023-11-01T04:17:05.957 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.200682834 2023-11-01T04:17:05.957 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.957 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.958 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.958 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.200699+0000", 2023-11-01T04:17:05.958 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000228743 2023-11-01T04:17:05.958 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.958 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.958 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.959 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.959 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.959 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:625 3.bs0 3:d1d3cdc7:::smithi19039307-26:head [rollback 12] snapc 27=[27,26,22,1b,19,18,12,11] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2023-11-01T04:17:05.959 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:54.789569+0000", 2023-11-01T04:17:05.960 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.000404856999999, 2023-11-01T04:17:05.960 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.412308589, 2023-11-01T04:17:05.960 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.960 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:05.960 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.960 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.961 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.961 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 625 2023-11-01T04:17:05.961 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.961 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.961 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.962 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.962 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789569+0000", 2023-11-01T04:17:05.962 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.962 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.962 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.962 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.963 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789569+0000", 2023-11-01T04:17:05.963 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.963 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.963 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.964 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789570+0000", 2023-11-01T04:17:05.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0299999999999998e-07 2023-11-01T04:17:05.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.964 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.964 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.965 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789574+0000", 2023-11-01T04:17:05.965 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5070000000000002e-06 2023-11-01T04:17:05.965 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.965 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.965 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.965 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789609+0000", 2023-11-01T04:17:05.966 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5046000000000003e-05 2023-11-01T04:17:05.966 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.966 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.966 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.966 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789622+0000", 2023-11-01T04:17:05.967 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3040000000000001e-05 2023-11-01T04:17:05.967 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.967 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.967 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.967 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.889948+0000", 2023-11-01T04:17:05.967 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10032589 2023-11-01T04:17:05.968 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.968 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.968 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:05.968 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.889971+0000", 2023-11-01T04:17:05.968 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2402999999999999e-05 2023-11-01T04:17:05.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.969 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.969 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.995630+0000", 2023-11-01T04:17:05.969 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10565901599999999 2023-11-01T04:17:05.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.970 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.970 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.970 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.995814+0000", 2023-11-01T04:17:05.970 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018381699999999999 2023-11-01T04:17:05.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.971 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.971 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:05.971 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.996939+0000", 2023-11-01T04:17:05.971 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011253089999999999 2023-11-01T04:17:05.971 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.972 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.972 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:05.972 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.999791+0000", 2023-11-01T04:17:05.972 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028522349999999998 2023-11-01T04:17:05.972 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.972 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.973 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:05.973 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.201667+0000", 2023-11-01T04:17:05.973 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201875794 2023-11-01T04:17:05.973 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.973 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.973 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.974 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.201878+0000", 2023-11-01T04:17:05.974 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021072899999999999 2023-11-01T04:17:05.974 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.974 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.974 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.974 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.975 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.975 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:626 3.bs0 3:d1d3cdc7:::smithi19039307-26:head [append 0~0] snapc 27=[27,26,22,1b,19,18,12,11] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2023-11-01T04:17:05.975 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:54.789661+0000", 2023-11-01T04:17:05.975 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.000313244999999, 2023-11-01T04:17:05.975 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.51282840500000004, 2023-11-01T04:17:05.976 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.976 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:05.976 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.976 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.976 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.976 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 626 2023-11-01T04:17:05.977 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.977 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.977 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.977 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.977 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789661+0000", 2023-11-01T04:17:05.977 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.978 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.978 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.978 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.978 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789661+0000", 2023-11-01T04:17:05.978 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.979 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.979 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.979 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.979 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789661+0000", 2023-11-01T04:17:05.979 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3e-07 2023-11-01T04:17:05.979 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.980 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.980 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.980 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789665+0000", 2023-11-01T04:17:05.980 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8430000000000003e-06 2023-11-01T04:17:05.981 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.981 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.981 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.981 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789677+0000", 2023-11-01T04:17:05.981 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2057e-05 2023-11-01T04:17:05.981 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.982 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.982 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.982 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789688+0000", 2023-11-01T04:17:05.982 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0526000000000001e-05 2023-11-01T04:17:05.982 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.983 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.983 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.983 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.890065+0000", 2023-11-01T04:17:05.983 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100377709 2023-11-01T04:17:05.983 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.984 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.984 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:05.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.890076+0000", 2023-11-01T04:17:05.984 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0669999999999999e-05 2023-11-01T04:17:05.985 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.985 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.985 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.985 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.997793+0000", 2023-11-01T04:17:05.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.107716457 2023-11-01T04:17:05.985 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.986 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.986 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:05.986 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.997961+0000", 2023-11-01T04:17:05.986 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000168064 2023-11-01T04:17:05.986 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.987 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:05.987 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.998972+0000", 2023-11-01T04:17:05.987 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001011184 2023-11-01T04:17:05.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.988 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.988 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:05.988 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.200758+0000", 2023-11-01T04:17:05.988 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201785986 2023-11-01T04:17:05.988 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.988 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.989 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:05.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.302355+0000", 2023-11-01T04:17:05.989 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10159710499999999 2023-11-01T04:17:05.989 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.989 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.990 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:05.990 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.302489+0000", 2023-11-01T04:17:05.990 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000134074 2023-11-01T04:17:05.990 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.990 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:05.991 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:05.991 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.991 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.991 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:714 3.4s0 3:2c54b66d:::smithi19039307-30:head [append 0~753664] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:05.991 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.518796+0000", 2023-11-01T04:17:05.992 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.2711780629999998, 2023-11-01T04:17:05.992 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40318051900000001, 2023-11-01T04:17:05.992 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:05.992 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:05.992 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:05.992 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:05.993 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:05.993 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 714 2023-11-01T04:17:05.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.993 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:05.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.993 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:05.994 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.518796+0000", 2023-11-01T04:17:05.994 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.994 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.994 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.994 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:05.995 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.518796+0000", 2023-11-01T04:17:05.995 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:05.995 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.995 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.995 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:05.995 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.518798+0000", 2023-11-01T04:17:05.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9099999999999999e-06 2023-11-01T04:17:05.996 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.996 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:05.996 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.519984+0000", 2023-11-01T04:17:05.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011858280000000001 2023-11-01T04:17:05.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:05.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.520019+0000", 2023-11-01T04:17:05.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5592000000000001e-05 2023-11-01T04:17:05.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.998 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:05.998 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.520047+0000", 2023-11-01T04:17:05.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7915999999999998e-05 2023-11-01T04:17:05.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:05.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:05.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.520190+0000", 2023-11-01T04:17:05.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000142473 2023-11-01T04:17:05.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:05.999 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:06.000 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.520401+0000", 2023-11-01T04:17:06.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021150800000000001 2023-11-01T04:17:06.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:06.001 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.522144+0000", 2023-11-01T04:17:06.001 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017434130000000001 2023-11-01T04:17:06.001 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.001 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.001 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:06.002 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.528429+0000", 2023-11-01T04:17:06.002 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006284258 2023-11-01T04:17:06.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.002 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.002 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:06.002 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.921854+0000", 2023-11-01T04:17:06.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.39342484900000002 2023-11-01T04:17:06.003 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.003 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:06.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.921976+0000", 2023-11-01T04:17:06.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000122772 2023-11-01T04:17:06.004 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:06.004 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:06.004 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:06.004 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.004 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.005 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:715 3.4s0 3:2c54b66d:::smithi19039307-30:head [append 0~688128] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:06.005 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.520089+0000", 2023-11-01T04:17:06.005 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.2698847349999998, 2023-11-01T04:17:06.005 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40421347800000001, 2023-11-01T04:17:06.005 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:06.005 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:06.006 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:06.006 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:06.006 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:06.006 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 715 2023-11-01T04:17:06.006 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.006 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:06.007 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.007 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:06.007 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.520089+0000", 2023-11-01T04:17:06.007 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:06.007 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.008 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.008 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:06.008 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.520089+0000", 2023-11-01T04:17:06.008 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:06.008 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.008 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.009 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:06.009 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.520091+0000", 2023-11-01T04:17:06.009 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8199999999999999e-06 2023-11-01T04:17:06.009 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.009 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.010 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:06.010 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.521225+0000", 2023-11-01T04:17:06.010 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001134003 2023-11-01T04:17:06.010 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.010 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.011 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:06.011 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.521257+0000", 2023-11-01T04:17:06.011 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2496000000000002e-05 2023-11-01T04:17:06.011 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.011 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.011 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:06.012 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.521279+0000", 2023-11-01T04:17:06.012 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1869e-05 2023-11-01T04:17:06.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.012 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.012 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:06.013 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.523440+0000", 2023-11-01T04:17:06.013 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002161086 2023-11-01T04:17:06.013 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.013 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.013 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:06.013 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.523762+0000", 2023-11-01T04:17:06.014 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000321155 2023-11-01T04:17:06.014 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.014 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.014 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:06.014 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.526059+0000", 2023-11-01T04:17:06.014 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022970780000000001 2023-11-01T04:17:06.015 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.015 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.015 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:06.015 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.529897+0000", 2023-11-01T04:17:06.015 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038381040000000002 2023-11-01T04:17:06.016 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.016 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.016 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:06.016 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.924231+0000", 2023-11-01T04:17:06.016 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.39433468999999999 2023-11-01T04:17:06.016 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.017 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.017 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:06.017 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.924303+0000", 2023-11-01T04:17:06.017 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1177e-05 2023-11-01T04:17:06.017 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:06.017 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:06.018 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:06.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.018 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:716 3.4s0 3:2c54b66d:::smithi19039307-30:head [append 0~475136] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:06.018 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.521315+0000", 2023-11-01T04:17:06.019 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.2686589509999999, 2023-11-01T04:17:06.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60362909399999998, 2023-11-01T04:17:06.019 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:06.019 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:06.019 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:06.019 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:06.020 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:06.020 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 716 2023-11-01T04:17:06.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.020 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:06.020 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.020 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:06.021 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.521315+0000", 2023-11-01T04:17:06.021 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:06.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.021 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.021 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:06.022 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.521315+0000", 2023-11-01T04:17:06.022 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:06.022 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.022 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.022 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:06.022 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.521317+0000", 2023-11-01T04:17:06.023 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0099999999999998e-06 2023-11-01T04:17:06.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.023 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:06.023 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.522189+0000", 2023-11-01T04:17:06.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00087234300000000003 2023-11-01T04:17:06.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.024 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:06.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.522232+0000", 2023-11-01T04:17:06.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3242999999999999e-05 2023-11-01T04:17:06.025 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.025 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.025 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:06.025 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622335+0000", 2023-11-01T04:17:06.025 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10010261500000001 2023-11-01T04:17:06.025 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.026 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.026 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:06.026 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622538+0000", 2023-11-01T04:17:06.026 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020300799999999999 2023-11-01T04:17:06.026 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.027 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.027 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:06.027 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622766+0000", 2023-11-01T04:17:06.027 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022807099999999999 2023-11-01T04:17:06.027 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.027 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.028 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:06.028 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.624385+0000", 2023-11-01T04:17:06.028 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001619073 2023-11-01T04:17:06.028 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.028 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.029 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:06.029 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.627002+0000", 2023-11-01T04:17:06.029 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026171440000000001 2023-11-01T04:17:06.029 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.029 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:06.030 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.124869+0000", 2023-11-01T04:17:06.030 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.49786697400000002 2023-11-01T04:17:06.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.031 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.031 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:06.031 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.124944+0000", 2023-11-01T04:17:06.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4612999999999999e-05 2023-11-01T04:17:06.031 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:06.031 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:06.032 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:06.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.032 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:717 3.4s0 3:2c54b66d:::smithi19039307-30:head [setxattr _header (57) in=7b] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:06.032 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.622376+0000", 2023-11-01T04:17:06.032 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.1675979409999999, 2023-11-01T04:17:06.033 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50279286700000003, 2023-11-01T04:17:06.033 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:06.033 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:06.033 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:06.033 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:06.033 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:06.034 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 717 2023-11-01T04:17:06.034 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.034 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:06.034 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.034 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:06.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622376+0000", 2023-11-01T04:17:06.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:06.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.035 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:06.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622376+0000", 2023-11-01T04:17:06.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:06.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:06.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622378+0000", 2023-11-01T04:17:06.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9400000000000001e-06 2023-11-01T04:17:06.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:06.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622388+0000", 2023-11-01T04:17:06.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9629999999999993e-06 2023-11-01T04:17:06.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:06.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622407+0000", 2023-11-01T04:17:06.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8692999999999999e-05 2023-11-01T04:17:06.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:06.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622417+0000", 2023-11-01T04:17:06.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0966e-05 2023-11-01T04:17:06.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.040 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:06.040 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.625681+0000", 2023-11-01T04:17:06.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032634399999999998 2023-11-01T04:17:06.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.040 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:06.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.625865+0000", 2023-11-01T04:17:06.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018365800000000001 2023-11-01T04:17:06.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.041 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:06.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.626617+0000", 2023-11-01T04:17:06.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00075261700000000002 2023-11-01T04:17:06.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.042 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:06.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.627981+0000", 2023-11-01T04:17:06.043 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001364117 2023-11-01T04:17:06.043 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.043 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.043 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:06.043 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.125056+0000", 2023-11-01T04:17:06.043 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.49707428199999998 2023-11-01T04:17:06.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:06.044 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.125169+0000", 2023-11-01T04:17:06.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011319099999999999 2023-11-01T04:17:06.045 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:06.045 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:06.045 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:06.045 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.045 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.045 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:718 3.4s0 3:2c54b66d:::smithi19039307-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:06.046 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.622459+0000", 2023-11-01T04:17:06.046 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.1675143600000002, 2023-11-01T04:17:06.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.70913759799999998, 2023-11-01T04:17:06.046 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:06.046 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:06.046 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:06.047 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:06.047 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:06.047 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 718 2023-11-01T04:17:06.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.047 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:06.048 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.048 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:06.048 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622459+0000", 2023-11-01T04:17:06.048 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:06.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.048 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.049 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:06.049 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622459+0000", 2023-11-01T04:17:06.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:06.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:06.050 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622460+0000", 2023-11-01T04:17:06.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0299999999999997e-07 2023-11-01T04:17:06.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.051 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:06.051 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622465+0000", 2023-11-01T04:17:06.051 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5900000000000001e-06 2023-11-01T04:17:06.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.052 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:06.052 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622478+0000", 2023-11-01T04:17:06.052 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.295e-05 2023-11-01T04:17:06.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:06.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622487+0000", 2023-11-01T04:17:06.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7190000000000007e-06 2023-11-01T04:17:06.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:06.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.628970+0000", 2023-11-01T04:17:06.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006483234 2023-11-01T04:17:06.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:06.055 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.629139+0000", 2023-11-01T04:17:06.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016887599999999999 2023-11-01T04:17:06.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:06.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.125755+0000", 2023-11-01T04:17:06.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.496615957 2023-11-01T04:17:06.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.056 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:06.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.125921+0000", 2023-11-01T04:17:06.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000165818 2023-11-01T04:17:06.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.057 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:06.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.331597+0000", 2023-11-01T04:17:06.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20567655100000001 2023-11-01T04:17:06.058 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:06.058 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:06.058 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:06.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.059 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:742 3.4s0 3:22b97aa6:::smithi19039307-42:head [copy-from ver 30] snapc 30=[30,2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2023-11-01T04:17:06.059 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.911862+0000", 2023-11-01T04:17:06.059 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.878112089, 2023-11-01T04:17:06.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.43783851299999998, 2023-11-01T04:17:06.059 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:06.060 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:06.060 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:06.060 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:06.060 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:06.060 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 742 2023-11-01T04:17:06.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.061 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:06.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:06.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.911862+0000", 2023-11-01T04:17:06.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:06.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.062 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:06.062 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.911862+0000", 2023-11-01T04:17:06.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:06.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:06.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.911865+0000", 2023-11-01T04:17:06.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8679999999999999e-06 2023-11-01T04:17:06.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.064 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.064 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:06.064 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.911876+0000", 2023-11-01T04:17:06.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0903e-05 2023-11-01T04:17:06.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:06.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.911899+0000", 2023-11-01T04:17:06.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3924999999999999e-05 2023-11-01T04:17:06.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:06.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.911913+0000", 2023-11-01T04:17:06.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3716999999999999e-05 2023-11-01T04:17:06.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:06.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.126362+0000", 2023-11-01T04:17:06.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.21444924400000001 2023-11-01T04:17:06.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:06.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.126515+0000", 2023-11-01T04:17:06.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015307100000000001 2023-11-01T04:17:06.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:06.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.240082+0000", 2023-11-01T04:17:06.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.113566865 2023-11-01T04:17:06.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.069 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:06.069 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.244253+0000", 2023-11-01T04:17:06.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004171129 2023-11-01T04:17:06.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:06.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.349538+0000", 2023-11-01T04:17:06.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105284816 2023-11-01T04:17:06.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:06.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:06.071 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:06.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.349700+0000", 2023-11-01T04:17:06.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016197499999999999 2023-11-01T04:17:06.073 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:06.074 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:06.074 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:06.074 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:06.074 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:06.074 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:17:06.075 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-11-01T04:17:06.076 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:17:06.077 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-01T04:17:06.077 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-11-01T04:17:06.078 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:17:06.079 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-01T04:17:06.079 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-01T04:17:06.080 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:17:06.081 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-01T04:17:06.081 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:17:06.081 INFO:tasks.rados.rados.0.smithi190.stdout:306: done (14 left) 2023-11-01T04:17:06.082 INFO:tasks.rados.rados.0.smithi190.stdout:307: read oid 29 snap -1 2023-11-01T04:17:06.082 INFO:tasks.rados.rados.0.smithi190.stdout:307: expect deleted 2023-11-01T04:17:06.082 INFO:tasks.rados.rados.0.smithi190.stdout:308: snap_create 2023-11-01T04:17:06.082 INFO:tasks.rados.rados.0.smithi190.stdout:293: finishing write tid 1 to smithi19039307-30 2023-11-01T04:17:06.082 INFO:tasks.rados.rados.0.smithi190.stdout:293: finishing write tid 2 to smithi19039307-30 2023-11-01T04:17:06.083 INFO:tasks.rados.rados.0.smithi190.stdout:292: finishing rollback tid 0 to smithi19039307-33 2023-11-01T04:17:06.083 INFO:tasks.rados.rados.0.smithi190.stdout:292: finishing rollback tid 1 to smithi19039307-33 2023-11-01T04:17:06.083 INFO:tasks.rados.rados.0.smithi190.stdout:297: finishing copy_from racing read to smithi19039307-3 2023-11-01T04:17:06.083 INFO:tasks.rados.rados.0.smithi190.stdout:292: finishing rollback tid 2 to smithi19039307-33 2023-11-01T04:17:06.083 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 33 v 90 (ObjNum 15 snap 0 seq_num 15) dirty exists 2023-11-01T04:17:06.083 INFO:tasks.rados.rados.0.smithi190.stdout:293: finishing write tid 3 to smithi19039307-30 2023-11-01T04:17:06.084 INFO:tasks.rados.rados.0.smithi190.stdout:290: finishing write tid 3 to smithi19039307-15 2023-11-01T04:17:06.084 INFO:tasks.rados.rados.0.smithi190.stdout:290: finishing write tid 4 to smithi19039307-15 2023-11-01T04:17:06.084 INFO:tasks.rados.rados.0.smithi190.stdout:298: finishing write tid 1 to smithi19039307-46 2023-11-01T04:17:06.084 INFO:tasks.rados.rados.0.smithi190.stdout:298: finishing write tid 2 to smithi19039307-46 2023-11-01T04:17:06.084 INFO:tasks.rados.rados.0.smithi190.stdout:298: finishing write tid 3 to smithi19039307-46 2023-11-01T04:17:06.084 INFO:tasks.rados.rados.0.smithi190.stdout:293: finishing write tid 4 to smithi19039307-30 2023-11-01T04:17:06.085 INFO:tasks.rados.rados.0.smithi190.stdout:299: finishing write tid 2 to smithi19039307-31 2023-11-01T04:17:06.085 INFO:tasks.rados.rados.0.smithi190.stdout:288: finishing copy_from to smithi19039307-32 2023-11-01T04:17:06.085 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 32 v 14 (ObjNum 6 snap 0 seq_num 6) dirty exists 2023-11-01T04:17:06.085 INFO:tasks.rados.rados.0.smithi190.stdout:298: finishing write tid 4 to smithi19039307-46 2023-11-01T04:17:06.086 INFO:tasks.rados.rados.0.smithi190.stdout:297: finishing copy_from to smithi19039307-3 2023-11-01T04:17:06.086 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 3 v 64 (ObjNum 103 snap 21 seq_num 103) dirty exists 2023-11-01T04:17:06.086 INFO:tasks.rados.rados.0.smithi190.stdout:303: finishing copy_from racing read to smithi19039307-42 2023-11-01T04:17:06.086 INFO:tasks.rados.rados.0.smithi190.stdout:291: finishing write tid 1 to smithi19039307-49 2023-11-01T04:17:06.086 INFO:tasks.rados.rados.0.smithi190.stdout:291: finishing write tid 2 to smithi19039307-49 2023-11-01T04:17:06.087 INFO:tasks.rados.rados.0.smithi190.stdout:286: expect (ObjNum 98 snap 21 seq_num 98) 2023-11-01T04:17:06.087 INFO:tasks.rados.rados.0.smithi190.stdout:291: finishing write tid 3 to smithi19039307-49 2023-11-01T04:17:06.087 INFO:tasks.rados.rados.0.smithi190.stdout:291: finishing write tid 4 to smithi19039307-49 2023-11-01T04:17:06.087 INFO:tasks.rados.rados.0.smithi190.stdout:295: finishing rollback tid 0 to smithi19039307-8 2023-11-01T04:17:06.087 INFO:tasks.rados.rados.0.smithi190.stdout:289: expect (ObjNum 21 snap 0 seq_num 21) 2023-11-01T04:17:06.087 INFO:tasks.rados.rados.0.smithi190.stdout:293: finishing write tid 5 to smithi19039307-30 2023-11-01T04:17:06.088 INFO:tasks.rados.rados.0.smithi190.stdout:293: oid 30 updating version 0 to 58 2023-11-01T04:17:06.088 INFO:tasks.rados.rados.0.smithi190.stdout:293: oid 30 version 58 is already newer than 56 2023-11-01T04:17:06.088 INFO:tasks.rados.rados.0.smithi190.stdout:293: oid 30 updating version 58 to 59 2023-11-01T04:17:06.088 INFO:tasks.rados.rados.0.smithi190.stdout:293: oid 30 version 59 is already newer than 57 2023-11-01T04:17:06.088 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 30 v 59 (ObjNum 114 snap 25 seq_num 114) dirty exists 2023-11-01T04:17:06.088 INFO:tasks.rados.rados.0.smithi190.stdout:293: left oid 30 (ObjNum 114 snap 25 seq_num 114) 2023-11-01T04:17:06.089 INFO:tasks.rados.rados.0.smithi190.stdout:299: finishing write tid 3 to smithi19039307-31 2023-11-01T04:17:06.089 INFO:tasks.rados.rados.0.smithi190.stdout:299: finishing write tid 4 to smithi19039307-31 2023-11-01T04:17:06.089 INFO:tasks.rados.rados.0.smithi190.stdout:303: finishing copy_from to smithi19039307-42 2023-11-01T04:17:06.089 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 42 v 30 (ObjNum 89 snap 20 seq_num 89) dirty exists 2023-11-01T04:17:06.089 INFO:tasks.rados.rados.0.smithi190.stdout:304: finishing rollback tid 0 to smithi19039307-21 2023-11-01T04:17:06.090 INFO:tasks.rados.rados.0.smithi190.stdout:290: finishing write tid 5 to smithi19039307-15 2023-11-01T04:17:06.090 INFO:tasks.rados.rados.0.smithi190.stdout:290: oid 15 updating version 0 to 54 2023-11-01T04:17:06.090 INFO:tasks.rados.rados.0.smithi190.stdout:290: oid 15 version 54 is already newer than 53 2023-11-01T04:17:06.090 INFO:tasks.rados.rados.0.smithi190.stdout:290: oid 15 version 54 is already newer than 52 2023-11-01T04:17:06.090 INFO:tasks.rados.rados.0.smithi190.stdout:290: oid 15 updating version 54 to 55 2023-11-01T04:17:06.090 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 15 v 55 (ObjNum 112 snap 25 seq_num 112) dirty exists 2023-11-01T04:17:06.091 INFO:tasks.rados.rados.0.smithi190.stdout:290: left oid 15 (ObjNum 112 snap 25 seq_num 112) 2023-11-01T04:17:06.091 INFO:tasks.rados.rados.0.smithi190.stdout:295: finishing rollback tid 1 to smithi19039307-8 2023-11-01T04:17:06.091 INFO:tasks.rados.rados.0.smithi190.stdout:295: finishing rollback tid 2 to smithi19039307-8 2023-11-01T04:17:06.091 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 8 v 98 (ObjNum 108 snap 24 seq_num 108) dirty exists 2023-11-01T04:17:06.091 INFO:tasks.rados.rados.0.smithi190.stdout:304: finishing rollback tid 1 to smithi19039307-21 2023-11-01T04:17:06.092 INFO:tasks.rados.rados.0.smithi190.stdout:304: finishing rollback tid 2 to smithi19039307-21 2023-11-01T04:17:06.092 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 21 v 32 (ObjNum 59 snap 5 seq_num 59) dirty exists 2023-11-01T04:17:06.092 INFO:tasks.rados.rados.0.smithi190.stdout:300: expect (ObjNum 11 snap 0 seq_num 11) 2023-11-01T04:17:06.092 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-01T04:17:06.093 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:17:06.093 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-01T04:17:06.093 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.12 2023-11-01T04:17:06.094 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:17:06.094 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-01T04:17:06.094 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:17:06.095 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-01T04:17:06.095 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:17:06.096 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-01T04:17:06.097 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-01T04:17:06.097 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-01T04:17:06.097 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:17:06.075+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:17:06.098 INFO:tasks.rados.rados.0.smithi190.stdout:299: finishing write tid 5 to smithi19039307-31 2023-11-01T04:17:06.098 INFO:tasks.rados.rados.0.smithi190.stdout:291: finishing write tid 5 to smithi19039307-49 2023-11-01T04:17:06.098 INFO:tasks.rados.rados.0.smithi190.stdout:291: oid 49 updating version 0 to 141 2023-11-01T04:17:06.098 INFO:tasks.rados.rados.0.smithi190.stdout:291: oid 49 version 141 is already newer than 138 2023-11-01T04:17:06.099 INFO:tasks.rados.rados.0.smithi190.stdout:291: oid 49 version 141 is already newer than 140 2023-11-01T04:17:06.099 INFO:tasks.rados.rados.0.smithi190.stdout:291: oid 49 version 141 is already newer than 139 2023-11-01T04:17:06.099 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 49 v 141 (ObjNum 113 snap 25 seq_num 113) dirty exists 2023-11-01T04:17:06.099 INFO:tasks.rados.rados.0.smithi190.stdout:291: left oid 49 (ObjNum 113 snap 25 seq_num 113) 2023-11-01T04:17:06.099 INFO:tasks.rados.rados.0.smithi190.stdout:299: finishing write tid 6 to smithi19039307-31 2023-11-01T04:17:06.099 INFO:tasks.rados.rados.0.smithi190.stdout:299: oid 31 updating version 0 to 67 2023-11-01T04:17:06.100 INFO:tasks.rados.rados.0.smithi190.stdout:299: oid 31 version 67 is already newer than 63 2023-11-01T04:17:06.100 INFO:tasks.rados.rados.0.smithi190.stdout:299: oid 31 version 67 is already newer than 65 2023-11-01T04:17:06.100 INFO:tasks.rados.rados.0.smithi190.stdout:299: oid 31 version 67 is already newer than 66 2023-11-01T04:17:06.100 INFO:tasks.rados.rados.0.smithi190.stdout:299: oid 31 version 67 is already newer than 64 2023-11-01T04:17:06.100 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 31 v 67 (ObjNum 116 snap 25 seq_num 116) dirty exists 2023-11-01T04:17:06.101 INFO:tasks.rados.rados.0.smithi190.stdout:299: left oid 31 (ObjNum 116 snap 25 seq_num 116) 2023-11-01T04:17:06.101 INFO:tasks.rados.rados.0.smithi190.stdout:298: finishing write tid 5 to smithi19039307-46 2023-11-01T04:17:06.101 INFO:tasks.rados.rados.0.smithi190.stdout:298: oid 46 updating version 0 to 33 2023-11-01T04:17:06.101 INFO:tasks.rados.rados.0.smithi190.stdout:298: oid 46 updating version 33 to 36 2023-11-01T04:17:06.101 INFO:tasks.rados.rados.0.smithi190.stdout:298: oid 46 version 36 is already newer than 34 2023-11-01T04:17:06.101 INFO:tasks.rados.rados.0.smithi190.stdout:298: oid 46 version 36 is already newer than 35 2023-11-01T04:17:06.102 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 46 v 36 (ObjNum 115 snap 25 seq_num 115) dirty exists 2023-11-01T04:17:06.102 INFO:tasks.rados.rados.0.smithi190.stdout:298: left oid 46 (ObjNum 115 snap 25 seq_num 115) 2023-11-01T04:17:06.176 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:17:06.174+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:17:06.251 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:06.251 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:06.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-01T04:17:06.308 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:06.306+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:06.378 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:06.376+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:06.427 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:06.427 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:06.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-01T04:17:06.492 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:17:06.490+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:17:06.565 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:06.565 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:06.579 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:17:06.579+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:17:06.581 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-11-01T04:17:06.680 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:17:06.678+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:17:06.712 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:06.713 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:06.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-11-01T04:17:06.780 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:06.844 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:06.844 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:06.858 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-11-01T04:17:06.882 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:17:06.879+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:17:06.922 INFO:teuthology.orchestra.run.smithi153.stderr:noscrub is set 2023-11-01T04:17:06.930 INFO:tasks.rados.rados.0.smithi190.stdout:286: done (15 left) 2023-11-01T04:17:06.930 INFO:tasks.rados.rados.0.smithi190.stdout:288: done (14 left) 2023-11-01T04:17:06.930 INFO:tasks.rados.rados.0.smithi190.stdout:289: done (13 left) 2023-11-01T04:17:06.931 INFO:tasks.rados.rados.0.smithi190.stdout:290: done (12 left) 2023-11-01T04:17:06.931 INFO:tasks.rados.rados.0.smithi190.stdout:291: done (11 left) 2023-11-01T04:17:06.931 INFO:tasks.rados.rados.0.smithi190.stdout:292: done (10 left) 2023-11-01T04:17:06.931 INFO:tasks.rados.rados.0.smithi190.stdout:293: done (9 left) 2023-11-01T04:17:06.931 INFO:tasks.rados.rados.0.smithi190.stdout:295: done (8 left) 2023-11-01T04:17:06.932 INFO:tasks.rados.rados.0.smithi190.stdout:297: done (7 left) 2023-11-01T04:17:06.932 INFO:tasks.rados.rados.0.smithi190.stdout:298: done (6 left) 2023-11-01T04:17:06.932 INFO:tasks.rados.rados.0.smithi190.stdout:299: done (5 left) 2023-11-01T04:17:06.932 INFO:tasks.rados.rados.0.smithi190.stdout:300: done (4 left) 2023-11-01T04:17:06.932 INFO:tasks.rados.rados.0.smithi190.stdout:303: done (3 left) 2023-11-01T04:17:06.932 INFO:tasks.rados.rados.0.smithi190.stdout:304: done (2 left) 2023-11-01T04:17:06.933 INFO:tasks.rados.rados.0.smithi190.stdout:307: done (1 left) 2023-11-01T04:17:06.933 INFO:tasks.rados.rados.0.smithi190.stdout:308: done (0 left) 2023-11-01T04:17:06.933 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 33 current snap is 27 2023-11-01T04:17:06.933 INFO:tasks.rados.rados.0.smithi190.stdout:309: seq_num 117 ranges {262144=425984,688128=606208,1294336=540672,1835008=409600} 2023-11-01T04:17:06.935 INFO:tasks.rados.rados.0.smithi190.stdout:309: writing smithi19039307-33 from 262144 to 688128 tid 1 2023-11-01T04:17:06.940 INFO:tasks.rados.rados.0.smithi190.stdout:309: writing smithi19039307-33 from 688128 to 1294336 tid 2 2023-11-01T04:17:06.944 INFO:tasks.rados.rados.0.smithi190.stdout:309: writing smithi19039307-33 from 1294336 to 1835008 tid 3 2023-11-01T04:17:06.947 INFO:tasks.rados.rados.0.smithi190.stdout:309: writing smithi19039307-33 from 1835008 to 2244608 tid 4 2023-11-01T04:17:06.948 INFO:tasks.rados.rados.0.smithi190.stdout:310: copy_from oid 18 from oid 13 current snap is 27 2023-11-01T04:17:06.948 INFO:tasks.rados.rados.0.smithi190.stdout:311: snap_create 2023-11-01T04:17:06.983 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:06.983 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:06.983 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:17:06.980+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:17:06.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-01T04:17:07.083 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:07.104 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:07.107 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:07.107 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-01T04:17:07.151 INFO:tasks.rados.rados.0.smithi190.stdout:309: finishing write tid 1 to smithi19039307-33 2023-11-01T04:17:07.185 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:17:07.182+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:17:07.215 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:07.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-01T04:17:07.253 INFO:tasks.rados.rados.0.smithi190.stdout:309: finishing write tid 2 to smithi19039307-33 2023-11-01T04:17:07.275 INFO:tasks.rados.rados.0.smithi190.stdout:310: finishing copy_from racing read to smithi19039307-18 2023-11-01T04:17:07.286 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:17:07.283+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:17:07.361 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:07.364 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:07.364 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-01T04:17:07.376 INFO:tasks.rados.rados.0.smithi190.stdout:310: finishing copy_from to smithi19039307-18 2023-11-01T04:17:07.376 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 18 v 78 (ObjNum 98 snap 21 seq_num 98) dirty exists 2023-11-01T04:17:07.387 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:07.384+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:07.488 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:17:07.486+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:17:07.498 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:07.499 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:07.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-01T04:17:07.588 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:07.584+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:07.636 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:07.636 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:07.650 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-01T04:17:07.662 INFO:tasks.rados.rados.0.smithi190.stdout:309: finishing write tid 3 to smithi19039307-33 2023-11-01T04:17:07.662 INFO:tasks.rados.rados.0.smithi190.stdout:309: finishing write tid 4 to smithi19039307-33 2023-11-01T04:17:07.663 INFO:tasks.rados.rados.0.smithi190.stdout:309: finishing write tid 5 to smithi19039307-33 2023-11-01T04:17:07.690 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:17:07.687+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:17:07.769 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:07.769 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:07.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-01T04:17:07.790 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:07.892 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:17:07.893+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:17:07.941 INFO:tasks.rados.rados.0.smithi190.stdout:310: done (2 left) 2023-11-01T04:17:07.941 INFO:tasks.rados.rados.0.smithi190.stdout:311: done (1 left) 2023-11-01T04:17:07.942 INFO:tasks.rados.rados.0.smithi190.stdout:312: snap_create 2023-11-01T04:17:07.948 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:07.948 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:07.963 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-01T04:17:07.992 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:17:07.991+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:17:08.065 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2023-11-01T04:17:08.067 INFO:tasks.rados.rados.0.smithi190.stdout:309: finishing write tid 6 to smithi19039307-33 2023-11-01T04:17:08.067 INFO:tasks.rados.rados.0.smithi190.stdout:309: oid 33 updating version 0 to 98 2023-11-01T04:17:08.067 INFO:tasks.rados.rados.0.smithi190.stdout:309: oid 33 version 98 is already newer than 96 2023-11-01T04:17:08.068 INFO:tasks.rados.rados.0.smithi190.stdout:309: oid 33 version 98 is already newer than 95 2023-11-01T04:17:08.068 INFO:tasks.rados.rados.0.smithi190.stdout:309: oid 33 version 98 is already newer than 97 2023-11-01T04:17:08.068 INFO:tasks.rados.rados.0.smithi190.stdout:309: oid 33 version 98 is already newer than 94 2023-11-01T04:17:08.068 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 33 v 98 (ObjNum 117 snap 27 seq_num 117) dirty exists 2023-11-01T04:17:08.068 INFO:tasks.rados.rados.0.smithi190.stdout:309: left oid 33 (ObjNum 117 snap 27 seq_num 117) 2023-11-01T04:17:08.092 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:08.098 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-01T04:17:08.125 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:08.125 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:08.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-01T04:17:08.184 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:08.187 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['config', 'set', 'filestore_inject_stall', '3'] 2023-11-01T04:17:08.194 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:08.190+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:08.255 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:17:08.256 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-11-01T04:17:08.263 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:08.263 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:08.276 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:17:08.276 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-01T04:17:08.278 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-01T04:17:08.295 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:08.292+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:08.296 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:17:08.296 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-01T04:17:08.337 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:17:08.338 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-01T04:17:08.354 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:17:08.355 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-01T04:17:08.375 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:17:08.375 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-01T04:17:08.392 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:17:08.393 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:17:08.393 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2023-11-01T04:17:08.393 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:17:08.394 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:08.397 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:17:08.397 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-01T04:17:08.398 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-11-01T04:17:08.407 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-01T04:17:08.415 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:17:08.416 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-01T04:17:08.437 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:17:08.438 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-01T04:17:08.460 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:17:08.460 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-01T04:17:08.480 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:17:08.481 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-01T04:17:08.481 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.12 2023-11-01T04:17:08.497 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:08.494+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:08.501 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:17:08.501 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-01T04:17:08.523 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:17:08.524 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-01T04:17:08.533 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:17:08.533 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:17:08.534 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2023-11-01T04:17:08.534 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2023-11-01T04:17:08.534 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:17:08.545 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:17:08.546 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-01T04:17:08.547 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-01T04:17:08.549 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-01T04:17:08.550 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-01T04:17:08.597 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:08.596+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:08.699 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:08.696+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:08.719 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:17:08.719 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2023-11-01T04:17:08.719 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2023-11-01T04:17:08.720 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:17:08.720 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.720 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:188 3.fs0 3:f81e983c:::smithi19039307-34:head [create,setxattr XNeZS[lOPtVca\\Pa[\\rdfbinb[bcjhtYSpmcafZm (1928) in=40b,setxattr TfflYYqPnps]lo^qSlp___TitNdd`]kaPTVOTcsh (1648) in=40b,setxattr TNb_onpY[j_dkhfmnWtm]Yktg]]RNtbQe[OUTNlX (770) in=40b,setxattr ZcgnkhY\\orTYaZh]PPldj`SNq`p\\cX`k]h\\bZY_Y (700) in=40b,setxattr gPQVnqhpsVhW_kq`QnafXrlRZtZ\\YnRftgQTishR (595) in=40b,setxattr U`XqYi[WhoqnpSPRc\\RhnppZp^qY\\qUfjpqs[r^a (1892) in=40b,setxattr cTmhd`pWgrRf_VfVgOtgeOP^ShP]lsOtpoTj[\\gl (418) in=40b,setxattr aa_taYTfslg`lRRrOkkraN\\XiW^V\\ljl]NqXin`a (314) in=40b,setxattr ibl]`igtgVThY^Qse\\okYYUtTmolX\\W]p]Q[P\\il (1403) in=40b,setxattr \\ZabSO[UUqeZa`hddNh[ZjaRRgUZoQtkn\\QdsX]W (1541) in=40b,setxattr jUkQUPTNbo`rZNmYlUmhb]ojc]aeheoVRfp]bPWf (5052) in=40b,setxattr Ofj`oQ^cR]da^SPsTaXmmPse\\OZ`VSqQ]U`efsZl (419) in=40b,setxattr a[cdkWos\\OrUVPciUaa^_^`lWVTScffiWkd\\[kgf (1261) in=40b,setxattr jXb]\\WhbatdZ[qTTiZO]kPd^`l[kajOYiQ[iqSYc (244) in=40b,setxattr \\im]jQhbanUn\\_^h`jaU]W`ReaZN_je[]kgg (1578) in=36b,setxattr ZbosoRORS^_de]rN^_daOiha\\Q\\eRae^[sQUjXsV (9120) in=40b,setxattr gcn]XpZUkOZr\\pkrVVZpYiR\\alh_PaZa_S[PYcn] (384) in=40b,setxattr \\^qUOaNXSTjtmafhfbo\\gseTTkV`mgnacYq^bkcc (34) in=40b,setxattr iUa^mOQ]TqjVSUU_nXfY]f`NjljiiaYbrm[o]aes (1611) in=40b,setxattr tXaQPQbc^aiV[^reVOPpib^caoWf^XgqilO^_g[k (1016) in=40b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:08.720 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:16.964946+0000", 2023-11-01T04:17:08.720 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 51.753447360999999, 2023-11-01T04:17:08.721 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.309617849, 2023-11-01T04:17:08.721 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.721 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:08.721 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:08.721 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:08.722 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:08.722 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 188 2023-11-01T04:17:08.722 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.722 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.722 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.723 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.726 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.964946+0000", 2023-11-01T04:17:08.726 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.726 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.726 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.727 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.727 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.964946+0000", 2023-11-01T04:17:08.727 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.727 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.727 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.728 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.728 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.964948+0000", 2023-11-01T04:17:08.728 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.824e-06 2023-11-01T04:17:08.728 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.728 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.728 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.729 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.965026+0000", 2023-11-01T04:17:08.729 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.8151000000000004e-05 2023-11-01T04:17:08.729 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.729 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.729 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.729 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.965047+0000", 2023-11-01T04:17:08.730 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0891e-05 2023-11-01T04:17:08.730 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.730 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.730 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.730 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.965062+0000", 2023-11-01T04:17:08.731 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5310000000000001e-05 2023-11-01T04:17:08.731 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.731 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.731 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.731 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.965154+0000", 2023-11-01T04:17:08.731 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2163999999999994e-05 2023-11-01T04:17:08.732 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.732 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.732 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2023-11-01T04:17:08.732 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:16.965165+0000", 2023-11-01T04:17:08.732 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0703000000000001e-05 2023-11-01T04:17:08.733 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.733 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.733 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.733 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.167756+0000", 2023-11-01T04:17:08.733 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20259100399999999 2023-11-01T04:17:08.734 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.734 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.734 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:08.734 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.167952+0000", 2023-11-01T04:17:08.734 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00019548100000000001 2023-11-01T04:17:08.735 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.735 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.735 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:08.735 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.169709+0000", 2023-11-01T04:17:08.735 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0017572989999999999 2023-11-01T04:17:08.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.736 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.736 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:08.736 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.171668+0000", 2023-11-01T04:17:08.736 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0019592659999999999 2023-11-01T04:17:08.736 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.737 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.737 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:08.737 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.274435+0000", 2023-11-01T04:17:08.737 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10276656000000001 2023-11-01T04:17:08.737 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.737 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.274564+0000", 2023-11-01T04:17:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012919599999999999 2023-11-01T04:17:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.738 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 37/31 ECSubWriteReply(tid=9, last_complete=37'4, committed=1, applied=1))", 2023-11-01T04:17:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:17.277556+0000", 2023-11-01T04:17:08.739 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 51.440836996000002, 2023-11-01T04:17:08.740 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20033701600000001, 2023-11-01T04:17:08.740 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.740 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:08.740 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.740 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.741 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.741 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.277556+0000", 2023-11-01T04:17:08.741 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.741 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.741 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.742 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.742 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.277556+0000", 2023-11-01T04:17:08.742 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.742 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.742 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.742 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.743 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.277558+0000", 2023-11-01T04:17:08.743 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.217e-06 2023-11-01T04:17:08.743 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.743 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.743 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.743 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.277571+0000", 2023-11-01T04:17:08.744 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3032e-05 2023-11-01T04:17:08.744 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.744 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.744 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.744 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.277593+0000", 2023-11-01T04:17:08.744 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2160000000000001e-05 2023-11-01T04:17:08.745 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.745 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.745 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.745 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.377685+0000", 2023-11-01T04:17:08.745 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10009243700000001 2023-11-01T04:17:08.745 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.746 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.746 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.746 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.477859+0000", 2023-11-01T04:17:08.746 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10017401500000001 2023-11-01T04:17:08.746 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.747 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.747 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.747 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.477894+0000", 2023-11-01T04:17:08.747 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4155000000000001e-05 2023-11-01T04:17:08.747 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.747 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.748 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.748 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.748 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.748 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(osd.11.22:2 3.cs0 3:33c6488a:::smithi19039307-6:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:08.748 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:17.378858+0000", 2023-11-01T04:17:08.749 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 51.339535605000002, 2023-11-01T04:17:08.749 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.203072057, 2023-11-01T04:17:08.749 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.749 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:08.749 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.750 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.750 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.750 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378858+0000", 2023-11-01T04:17:08.750 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.750 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.751 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.751 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.751 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378858+0000", 2023-11-01T04:17:08.751 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.751 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.751 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.752 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.752 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378860+0000", 2023-11-01T04:17:08.752 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5339999999999998e-06 2023-11-01T04:17:08.752 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.752 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.753 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.753 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378872+0000", 2023-11-01T04:17:08.753 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1365999999999999e-05 2023-11-01T04:17:08.753 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.753 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.753 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.754 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378896+0000", 2023-11-01T04:17:08.754 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3935999999999999e-05 2023-11-01T04:17:08.754 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.754 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.754 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.754 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.378909+0000", 2023-11-01T04:17:08.755 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3523000000000001e-05 2023-11-01T04:17:08.755 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.755 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.755 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.755 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.379027+0000", 2023-11-01T04:17:08.755 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000117963 2023-11-01T04:17:08.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.756 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.756 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:08.756 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.379186+0000", 2023-11-01T04:17:08.756 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000158939 2023-11-01T04:17:08.756 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.757 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.757 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.757 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.581930+0000", 2023-11-01T04:17:08.757 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.202743796 2023-11-01T04:17:08.757 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.757 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.758 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.758 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.758 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.758 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(osd.14.22:2 3.cs0 3:33c6488a:::smithi19039307-6:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:17:08.758 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.134094+0000", 2023-11-01T04:17:08.759 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 42.584299260999998, 2023-11-01T04:17:08.759 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.35681233299999998, 2023-11-01T04:17:08.759 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.759 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:08.759 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.760 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.760 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.760 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134094+0000", 2023-11-01T04:17:08.760 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.760 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.761 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.761 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.761 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134094+0000", 2023-11-01T04:17:08.761 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.761 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.762 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.762 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.762 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134096+0000", 2023-11-01T04:17:08.762 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1919999999999999e-06 2023-11-01T04:17:08.762 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.763 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.763 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.763 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134108+0000", 2023-11-01T04:17:08.763 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1508000000000001e-05 2023-11-01T04:17:08.763 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.764 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.764 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.764 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134129+0000", 2023-11-01T04:17:08.764 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0968e-05 2023-11-01T04:17:08.764 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.765 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.765 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.765 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134151+0000", 2023-11-01T04:17:08.765 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2480000000000002e-05 2023-11-01T04:17:08.765 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.766 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.766 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.766 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134288+0000", 2023-11-01T04:17:08.766 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013673799999999999 2023-11-01T04:17:08.766 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.767 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.767 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:08.767 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.134380+0000", 2023-11-01T04:17:08.767 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.1555e-05 2023-11-01T04:17:08.767 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.767 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.768 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.768 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.490907+0000", 2023-11-01T04:17:08.768 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.35652689199999998 2023-11-01T04:17:08.768 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.768 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.769 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.769 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.769 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.769 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.cs0 45/31 ECSubReadReply(tid=18, attrs_read=0))", 2023-11-01T04:17:08.769 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.135072+0000", 2023-11-01T04:17:08.770 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 42.583321947000002, 2023-11-01T04:17:08.770 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20028368799999999, 2023-11-01T04:17:08.770 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.770 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:08.770 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.771 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.771 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.771 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135072+0000", 2023-11-01T04:17:08.771 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.771 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.771 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.772 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.772 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135072+0000", 2023-11-01T04:17:08.772 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.772 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.772 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.773 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.773 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135072+0000", 2023-11-01T04:17:08.773 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5199999999999995e-07 2023-11-01T04:17:08.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.773 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.773 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.774 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135109+0000", 2023-11-01T04:17:08.774 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6155999999999997e-05 2023-11-01T04:17:08.774 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.774 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.774 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.775 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135118+0000", 2023-11-01T04:17:08.775 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7399999999999999e-06 2023-11-01T04:17:08.775 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.775 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.775 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.775 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135123+0000", 2023-11-01T04:17:08.776 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.8160000000000001e-06 2023-11-01T04:17:08.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.776 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.776 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.776 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.335285+0000", 2023-11-01T04:17:08.777 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20016213599999999 2023-11-01T04:17:08.777 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.777 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.777 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.777 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.335355+0000", 2023-11-01T04:17:08.777 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9888000000000006e-05 2023-11-01T04:17:08.778 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.778 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.778 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.778 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.778 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.cs0 46/31 ECSubReadReply(tid=18, attrs_read=0))", 2023-11-01T04:17:08.779 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.135473+0000", 2023-11-01T04:17:08.779 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 42.582920115, 2023-11-01T04:17:08.779 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19999888900000001, 2023-11-01T04:17:08.779 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.779 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:08.780 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.780 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.780 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.780 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135473+0000", 2023-11-01T04:17:08.780 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.781 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.781 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.781 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.781 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135473+0000", 2023-11-01T04:17:08.781 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.782 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.782 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.782 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135474+0000", 2023-11-01T04:17:08.782 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1200000000000002e-07 2023-11-01T04:17:08.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.783 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.783 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.783 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135586+0000", 2023-11-01T04:17:08.783 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000112274 2023-11-01T04:17:08.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.783 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.784 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.784 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135602+0000", 2023-11-01T04:17:08.784 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5044e-05 2023-11-01T04:17:08.784 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.784 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.785 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.785 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.135611+0000", 2023-11-01T04:17:08.785 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0210000000000005e-06 2023-11-01T04:17:08.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.785 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.785 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.786 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.335430+0000", 2023-11-01T04:17:08.786 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19981938099999999 2023-11-01T04:17:08.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.786 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.786 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.787 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.335472+0000", 2023-11-01T04:17:08.787 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2357000000000001e-05 2023-11-01T04:17:08.787 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.787 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.787 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.788 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.788 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s2 46/31 ECSubWrite(tid=235, reqid=client.4559.0:329, at_version=46'35, trim_to=0'0, roll_forward_to=45'33))", 2023-11-01T04:17:08.788 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.179884+0000", 2023-11-01T04:17:08.788 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 42.538509048000002, 2023-11-01T04:17:08.788 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20567904100000001, 2023-11-01T04:17:08.788 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.789 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:08.789 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.789 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.789 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.789 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.179884+0000", 2023-11-01T04:17:08.789 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.790 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.790 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.790 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.179884+0000", 2023-11-01T04:17:08.790 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.791 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.791 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.791 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.179885+0000", 2023-11-01T04:17:08.791 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.74e-07 2023-11-01T04:17:08.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.792 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.792 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.180252+0000", 2023-11-01T04:17:08.792 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00036695899999999998 2023-11-01T04:17:08.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.793 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.793 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.180264+0000", 2023-11-01T04:17:08.793 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2850999999999999e-05 2023-11-01T04:17:08.793 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.793 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.794 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.794 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.280394+0000", 2023-11-01T04:17:08.794 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100129722 2023-11-01T04:17:08.794 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.794 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.795 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.795 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.280540+0000", 2023-11-01T04:17:08.795 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014556000000000001 2023-11-01T04:17:08.795 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.795 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.795 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:08.796 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.280557+0000", 2023-11-01T04:17:08.796 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7543000000000001e-05 2023-11-01T04:17:08.796 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.796 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.796 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:08.796 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.385509+0000", 2023-11-01T04:17:08.797 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104951378 2023-11-01T04:17:08.797 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.797 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.797 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.797 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.385563+0000", 2023-11-01T04:17:08.797 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4854e-05 2023-11-01T04:17:08.798 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.798 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.798 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.798 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.798 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.799 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.8s5 46/31 ECSubWrite(tid=30, reqid=client.4559.0:341, at_version=46'22, trim_to=0'0, roll_forward_to=43'14))", 2023-11-01T04:17:08.799 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.266862+0000", 2023-11-01T04:17:08.799 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 42.451531025999998, 2023-11-01T04:17:08.799 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20203428600000001, 2023-11-01T04:17:08.799 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.800 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:08.800 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.800 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.800 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.800 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266862+0000", 2023-11-01T04:17:08.800 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.801 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.801 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.801 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.801 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266862+0000", 2023-11-01T04:17:08.801 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.801 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.802 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.802 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.802 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266865+0000", 2023-11-01T04:17:08.802 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2840000000000001e-06 2023-11-01T04:17:08.802 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.803 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.803 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.803 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266897+0000", 2023-11-01T04:17:08.803 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2743000000000001e-05 2023-11-01T04:17:08.803 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.803 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.804 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.804 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266920+0000", 2023-11-01T04:17:08.804 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2878999999999999e-05 2023-11-01T04:17:08.804 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.804 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.805 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.805 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.266939+0000", 2023-11-01T04:17:08.805 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8185999999999998e-05 2023-11-01T04:17:08.805 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.805 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.805 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.806 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.467334+0000", 2023-11-01T04:17:08.806 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20039552699999999 2023-11-01T04:17:08.806 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.806 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.806 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:08.806 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.467350+0000", 2023-11-01T04:17:08.807 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5682999999999999e-05 2023-11-01T04:17:08.807 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.807 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.807 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:08.807 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.468855+0000", 2023-11-01T04:17:08.808 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001504825 2023-11-01T04:17:08.808 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.808 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.808 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.808 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.468897+0000", 2023-11-01T04:17:08.809 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2159000000000001e-05 2023-11-01T04:17:08.809 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.809 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.809 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.809 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.810 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.810 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:347 3.fs0 3:f81e983c:::smithi19039307-34:c [read 0~557056,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:17:08.810 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.032054+0000", 2023-11-01T04:17:08.810 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 41.686339324999999, 2023-11-01T04:17:08.810 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20755394999999999, 2023-11-01T04:17:08.811 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.811 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:08.811 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:08.811 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:08.811 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:08.812 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 347 2023-11-01T04:17:08.812 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.812 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.812 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.812 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.812 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032054+0000", 2023-11-01T04:17:08.813 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.813 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.813 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.813 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.813 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032054+0000", 2023-11-01T04:17:08.813 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.814 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.814 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.814 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.814 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032058+0000", 2023-11-01T04:17:08.814 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4740000000000001e-06 2023-11-01T04:17:08.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.815 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.815 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.815 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032074+0000", 2023-11-01T04:17:08.815 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.679e-05 2023-11-01T04:17:08.815 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.816 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.816 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.816 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032109+0000", 2023-11-01T04:17:08.816 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4309e-05 2023-11-01T04:17:08.816 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.816 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.817 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.817 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032128+0000", 2023-11-01T04:17:08.817 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8856999999999999e-05 2023-11-01T04:17:08.817 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.817 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.817 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.818 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032256+0000", 2023-11-01T04:17:08.818 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000128866 2023-11-01T04:17:08.818 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.818 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.818 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2023-11-01T04:17:08.818 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032268+0000", 2023-11-01T04:17:08.819 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1649e-05 2023-11-01T04:17:08.819 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.819 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.819 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.819 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.132806+0000", 2023-11-01T04:17:08.819 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10053780399999999 2023-11-01T04:17:08.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.820 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.820 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:08.820 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.132905+0000", 2023-11-01T04:17:08.820 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.9326000000000004e-05 2023-11-01T04:17:08.820 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.821 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.821 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.821 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.239608+0000", 2023-11-01T04:17:08.821 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.106702875 2023-11-01T04:17:08.821 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.822 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.822 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.822 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.822 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.822 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:348 3.fs0 3:f81e983c:::smithi19039307-34:c [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:17:08.822 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.032168+0000", 2023-11-01T04:17:08.823 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 41.686225030000003, 2023-11-01T04:17:08.823 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30608133399999998, 2023-11-01T04:17:08.823 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.823 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:08.823 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:08.824 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:08.824 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:08.824 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 348 2023-11-01T04:17:08.824 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.824 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.824 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.825 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.825 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032168+0000", 2023-11-01T04:17:08.825 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.825 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.825 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.825 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.826 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032168+0000", 2023-11-01T04:17:08.826 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.826 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.826 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.826 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.827 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032169+0000", 2023-11-01T04:17:08.827 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8199999999999999e-07 2023-11-01T04:17:08.827 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.827 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.827 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.827 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032174+0000", 2023-11-01T04:17:08.828 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1379999999999998e-06 2023-11-01T04:17:08.828 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.828 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.828 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.828 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032186+0000", 2023-11-01T04:17:08.829 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1229e-05 2023-11-01T04:17:08.829 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.829 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.829 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.829 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032193+0000", 2023-11-01T04:17:08.829 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2749999999999998e-06 2023-11-01T04:17:08.830 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.830 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.830 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.830 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032361+0000", 2023-11-01T04:17:08.830 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016853199999999999 2023-11-01T04:17:08.831 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.831 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.831 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:08.831 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032368+0000", 2023-11-01T04:17:08.831 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.686e-06 2023-11-01T04:17:08.832 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.832 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.832 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.832 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.135190+0000", 2023-11-01T04:17:08.832 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10282246 2023-11-01T04:17:08.833 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.833 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.833 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:08.833 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.135261+0000", 2023-11-01T04:17:08.833 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0296999999999995e-05 2023-11-01T04:17:08.834 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.834 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.834 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.834 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.338250+0000", 2023-11-01T04:17:08.834 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20298903500000001 2023-11-01T04:17:08.834 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.835 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.835 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.835 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.835 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.835 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:349 3.fs0 3:f81e983c:::smithi19039307-34:c [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:17:08.836 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.032204+0000", 2023-11-01T04:17:08.836 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 41.686189571, 2023-11-01T04:17:08.836 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.61100175800000001, 2023-11-01T04:17:08.836 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.836 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:08.836 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:08.837 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:08.837 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:08.837 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 349 2023-11-01T04:17:08.837 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.837 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.837 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.838 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.838 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032204+0000", 2023-11-01T04:17:08.838 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.838 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.838 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.838 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.839 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032204+0000", 2023-11-01T04:17:08.839 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.839 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.839 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.839 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032204+0000", 2023-11-01T04:17:08.840 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2200000000000004e-07 2023-11-01T04:17:08.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.840 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.840 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.840 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032209+0000", 2023-11-01T04:17:08.841 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5979999999999999e-06 2023-11-01T04:17:08.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.841 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.841 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.841 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032217+0000", 2023-11-01T04:17:08.841 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9019999999999999e-06 2023-11-01T04:17:08.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.842 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.842 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.842 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.032223+0000", 2023-11-01T04:17:08.842 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7180000000000001e-06 2023-11-01T04:17:08.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.843 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.843 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.843 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.132534+0000", 2023-11-01T04:17:08.843 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10031093200000001 2023-11-01T04:17:08.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.844 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.844 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:08.844 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.132550+0000", 2023-11-01T04:17:08.844 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6269999999999998e-05 2023-11-01T04:17:08.844 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.844 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.845 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.845 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.339795+0000", 2023-11-01T04:17:08.845 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20724526200000001 2023-11-01T04:17:08.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.845 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.845 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:08.846 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.340097+0000", 2023-11-01T04:17:08.846 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00030174899999999997 2023-11-01T04:17:08.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.846 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.846 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.847 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.643206+0000", 2023-11-01T04:17:08.847 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30310880499999998 2023-11-01T04:17:08.847 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.847 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.847 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.847 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.848 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.848 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.fs0 47/31 ECSubReadReply(tid=21, attrs_read=0))", 2023-11-01T04:17:08.848 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:27.349538+0000", 2023-11-01T04:17:08.848 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 41.368855576000001, 2023-11-01T04:17:08.848 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.29371594899999998, 2023-11-01T04:17:08.848 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.849 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:08.849 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.849 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.849 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.849 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349538+0000", 2023-11-01T04:17:08.849 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.850 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.850 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.850 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.850 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349538+0000", 2023-11-01T04:17:08.850 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.851 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.851 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.851 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349539+0000", 2023-11-01T04:17:08.851 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5370000000000001e-06 2023-11-01T04:17:08.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.852 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.852 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.852 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349659+0000", 2023-11-01T04:17:08.852 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011938 2023-11-01T04:17:08.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.853 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.853 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.853 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349686+0000", 2023-11-01T04:17:08.853 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.728e-05 2023-11-01T04:17:08.853 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.853 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.854 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.854 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.349702+0000", 2023-11-01T04:17:08.854 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5693e-05 2023-11-01T04:17:08.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.854 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.854 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.855 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.642479+0000", 2023-11-01T04:17:08.855 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.29277763699999998 2023-11-01T04:17:08.855 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.855 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.855 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.856 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:27.643254+0000", 2023-11-01T04:17:08.856 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00077442199999999998 2023-11-01T04:17:08.856 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.856 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.856 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.856 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.857 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.857 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(osd.13.22:6 3.fs0 3:f81e983c:::smithi19039307-34:head [assert-version v12,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:08.857 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:48.189226+0000", 2023-11-01T04:17:08.857 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 20.529167436000002, 2023-11-01T04:17:08.857 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30167290200000002, 2023-11-01T04:17:08.857 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.858 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:08.858 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.858 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.858 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.858 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.189226+0000", 2023-11-01T04:17:08.859 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.859 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.859 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.859 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.859 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.189226+0000", 2023-11-01T04:17:08.859 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.860 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.860 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.860 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.860 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.189229+0000", 2023-11-01T04:17:08.860 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4729999999999999e-06 2023-11-01T04:17:08.860 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.861 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.861 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.861 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.189239+0000", 2023-11-01T04:17:08.861 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0036e-05 2023-11-01T04:17:08.861 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.861 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.862 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.862 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.189260+0000", 2023-11-01T04:17:08.862 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1206999999999999e-05 2023-11-01T04:17:08.862 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.862 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.862 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.863 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.189277+0000", 2023-11-01T04:17:08.863 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6946e-05 2023-11-01T04:17:08.863 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.863 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.863 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.863 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.197102+0000", 2023-11-01T04:17:08.864 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0078254740000000007 2023-11-01T04:17:08.864 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.864 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.864 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:08.864 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.197404+0000", 2023-11-01T04:17:08.864 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00030222199999999999 2023-11-01T04:17:08.865 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.865 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.865 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.865 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.490899+0000", 2023-11-01T04:17:08.865 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.29349454400000002 2023-11-01T04:17:08.866 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.866 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.866 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.866 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.866 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.866 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s2 69/31 ECSubWrite(tid=324, reqid=client.4559.0:569, at_version=69'78, trim_to=0'0, roll_forward_to=68'74))", 2023-11-01T04:17:08.867 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:49.613254+0000", 2023-11-01T04:17:08.867 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 19.105139522000002, 2023-11-01T04:17:08.867 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20193297099999999, 2023-11-01T04:17:08.867 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.867 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:08.867 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.868 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.868 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.868 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613254+0000", 2023-11-01T04:17:08.868 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.868 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.868 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.869 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.869 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613254+0000", 2023-11-01T04:17:08.869 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.869 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.869 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.869 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.870 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613255+0000", 2023-11-01T04:17:08.870 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6700000000000003e-07 2023-11-01T04:17:08.870 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.870 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.870 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.871 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613466+0000", 2023-11-01T04:17:08.871 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00021175600000000001 2023-11-01T04:17:08.871 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.871 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.871 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.871 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613485+0000", 2023-11-01T04:17:08.872 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8499999999999999e-05 2023-11-01T04:17:08.872 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.872 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.872 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.872 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613495+0000", 2023-11-01T04:17:08.873 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7200000000000001e-06 2023-11-01T04:17:08.873 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.873 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.873 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.873 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.714261+0000", 2023-11-01T04:17:08.873 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10076626399999999 2023-11-01T04:17:08.874 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.874 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.874 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:08.874 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.714278+0000", 2023-11-01T04:17:08.874 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7462999999999999e-05 2023-11-01T04:17:08.875 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.875 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.875 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:08.875 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.815118+0000", 2023-11-01T04:17:08.875 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100840022 2023-11-01T04:17:08.875 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.876 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.876 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.876 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.815187+0000", 2023-11-01T04:17:08.876 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8678999999999998e-05 2023-11-01T04:17:08.876 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.877 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.877 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.877 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.877 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.877 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s2 69/31 ECSubWrite(tid=325, reqid=client.4559.0:570, at_version=69'79, trim_to=0'0, roll_forward_to=68'74))", 2023-11-01T04:17:08.877 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:49.613511+0000", 2023-11-01T04:17:08.878 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 19.104882899, 2023-11-01T04:17:08.878 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20304860599999999, 2023-11-01T04:17:08.878 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.878 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:08.878 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.878 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.879 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.879 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613511+0000", 2023-11-01T04:17:08.879 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.879 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.879 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.880 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.880 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613511+0000", 2023-11-01T04:17:08.880 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.880 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.880 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.880 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.881 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613511+0000", 2023-11-01T04:17:08.881 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0999999999999999e-07 2023-11-01T04:17:08.881 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.881 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.881 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.881 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613647+0000", 2023-11-01T04:17:08.882 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000136124 2023-11-01T04:17:08.882 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.882 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.882 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.882 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.613662+0000", 2023-11-01T04:17:08.883 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5146e-05 2023-11-01T04:17:08.883 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.883 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.883 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.883 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.713796+0000", 2023-11-01T04:17:08.884 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100133378 2023-11-01T04:17:08.884 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.884 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.884 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.884 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.814080+0000", 2023-11-01T04:17:08.884 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10028453499999999 2023-11-01T04:17:08.885 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.885 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.885 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:08.885 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.814096+0000", 2023-11-01T04:17:08.885 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6030000000000001e-05 2023-11-01T04:17:08.885 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.886 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.886 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:08.886 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.816487+0000", 2023-11-01T04:17:08.886 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002391121 2023-11-01T04:17:08.886 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.886 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.887 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.887 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.816559+0000", 2023-11-01T04:17:08.887 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1761999999999995e-05 2023-11-01T04:17:08.887 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.887 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.888 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.888 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.888 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.888 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.6s5 72/68 ECSubWrite(tid=102, reqid=client.4559.0:608, at_version=72'14, trim_to=0'0, roll_forward_to=0'0))", 2023-11-01T04:17:08.888 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:52.898759+0000", 2023-11-01T04:17:08.888 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 15.819634233, 2023-11-01T04:17:08.889 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30376394600000001, 2023-11-01T04:17:08.889 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.889 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:08.889 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.889 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.890 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.890 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.898759+0000", 2023-11-01T04:17:08.890 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.890 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.890 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.890 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.891 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.898759+0000", 2023-11-01T04:17:08.891 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.891 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.891 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.891 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.891 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.898760+0000", 2023-11-01T04:17:08.892 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0800000000000004e-07 2023-11-01T04:17:08.892 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.892 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.892 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.892 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.898794+0000", 2023-11-01T04:17:08.893 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.3899000000000002e-05 2023-11-01T04:17:08.893 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.893 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.893 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.893 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.898805+0000", 2023-11-01T04:17:08.893 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1467e-05 2023-11-01T04:17:08.894 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.894 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.894 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.894 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.898814+0000", 2023-11-01T04:17:08.894 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.2570000000000007e-06 2023-11-01T04:17:08.895 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.895 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.895 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.895 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.201072+0000", 2023-11-01T04:17:08.895 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30225848500000002 2023-11-01T04:17:08.895 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.896 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.896 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:08.896 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.201088+0000", 2023-11-01T04:17:08.896 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5549999999999999e-05 2023-11-01T04:17:08.896 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.897 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.897 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:08.897 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.202443+0000", 2023-11-01T04:17:08.897 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001355057 2023-11-01T04:17:08.897 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.897 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.898 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.898 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.202523+0000", 2023-11-01T04:17:08.898 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0422999999999999e-05 2023-11-01T04:17:08.898 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.898 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.899 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.899 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.899 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.899 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 72/31 ECSubWriteReply(tid=34, last_complete=72'20, committed=1, applied=1))", 2023-11-01T04:17:08.899 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:52.905947+0000", 2023-11-01T04:17:08.899 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 15.812446633, 2023-11-01T04:17:08.900 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20027774300000001, 2023-11-01T04:17:08.900 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.900 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:08.900 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.900 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.900 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.901 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.905947+0000", 2023-11-01T04:17:08.901 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.901 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.901 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.901 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.901 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.905947+0000", 2023-11-01T04:17:08.902 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.902 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.902 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.902 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.902 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.905948+0000", 2023-11-01T04:17:08.903 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5600000000000004e-07 2023-11-01T04:17:08.903 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.903 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.903 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.903 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.905952+0000", 2023-11-01T04:17:08.903 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6750000000000001e-06 2023-11-01T04:17:08.904 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.904 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.904 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.904 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.905965+0000", 2023-11-01T04:17:08.904 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2655e-05 2023-11-01T04:17:08.904 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.905 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.905 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.905 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:52.905972+0000", 2023-11-01T04:17:08.905 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6000000000000003e-06 2023-11-01T04:17:08.905 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.905 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.906 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.906 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.106189+0000", 2023-11-01T04:17:08.906 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.200217426 2023-11-01T04:17:08.906 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.906 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.907 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.907 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:53.106225+0000", 2023-11-01T04:17:08.907 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5531000000000003e-05 2023-11-01T04:17:08.907 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.907 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.907 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.908 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.908 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.908 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:659 3.cs0 3:33c6488a:::smithi19039307-6:head [copy-from ver 2] snapc 28=[28,26,22,19,18,12] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-11-01T04:17:08.908 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:58.843911+0000", 2023-11-01T04:17:08.908 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 9.8744823119999996, 2023-11-01T04:17:08.909 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.183501809, 2023-11-01T04:17:08.909 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.909 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:08.909 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:08.909 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:08.909 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:08.910 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 659 2023-11-01T04:17:08.910 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.910 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.910 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.910 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.910 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.843911+0000", 2023-11-01T04:17:08.911 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.911 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.911 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.911 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.911 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.843911+0000", 2023-11-01T04:17:08.911 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.912 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.912 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.912 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.912 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.843914+0000", 2023-11-01T04:17:08.912 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2760000000000001e-06 2023-11-01T04:17:08.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.913 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.913 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.913 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.843948+0000", 2023-11-01T04:17:08.913 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.3089000000000001e-05 2023-11-01T04:17:08.913 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.914 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.914 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.914 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.843982+0000", 2023-11-01T04:17:08.914 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.3961999999999998e-05 2023-11-01T04:17:08.914 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.915 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.915 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.915 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.844009+0000", 2023-11-01T04:17:08.915 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7498999999999999e-05 2023-11-01T04:17:08.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.915 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.916 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.916 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.247668+0000", 2023-11-01T04:17:08.916 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.40365936499999999 2023-11-01T04:17:08.916 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.916 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.916 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2023-11-01T04:17:08.917 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.247675+0000", 2023-11-01T04:17:08.917 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6699999999999997e-06 2023-11-01T04:17:08.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.917 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.917 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.918 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.182507+0000", 2023-11-01T04:17:08.918 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.93483153699999999 2023-11-01T04:17:08.918 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.918 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.918 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:08.919 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.182702+0000", 2023-11-01T04:17:08.919 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00019557 2023-11-01T04:17:08.919 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.919 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.919 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:08.919 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.823537+0000", 2023-11-01T04:17:08.920 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.64083475499999998 2023-11-01T04:17:08.920 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.920 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.920 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:08.920 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.825158+0000", 2023-11-01T04:17:08.920 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0016206199999999999 2023-11-01T04:17:08.921 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.921 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.921 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:08.921 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:01.027279+0000", 2023-11-01T04:17:08.921 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.202121202 2023-11-01T04:17:08.922 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.922 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.922 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.922 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:01.027413+0000", 2023-11-01T04:17:08.922 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013426399999999999 2023-11-01T04:17:08.922 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.923 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.923 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.923 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.923 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.923 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:660 3.cs0 3:33c6488a:::smithi19039307-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2023-11-01T04:17:08.923 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:58.844062+0000", 2023-11-01T04:17:08.924 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 9.8743318680000005, 2023-11-01T04:17:08.924 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.980937341, 2023-11-01T04:17:08.924 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.924 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:08.924 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:08.925 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:08.925 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:08.925 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 660 2023-11-01T04:17:08.925 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.925 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.926 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.926 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.926 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.844062+0000", 2023-11-01T04:17:08.926 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.927 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.927 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.927 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.844062+0000", 2023-11-01T04:17:08.927 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.927 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.928 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.928 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.928 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.844063+0000", 2023-11-01T04:17:08.928 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6029999999999999e-06 2023-11-01T04:17:08.929 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.929 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.929 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.929 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.844070+0000", 2023-11-01T04:17:08.929 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1960000000000002e-06 2023-11-01T04:17:08.930 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.930 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.930 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.930 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.844094+0000", 2023-11-01T04:17:08.930 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3159999999999998e-05 2023-11-01T04:17:08.931 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.931 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.931 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.931 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:58.844115+0000", 2023-11-01T04:17:08.931 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1685e-05 2023-11-01T04:17:08.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.932 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.932 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.932 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.447803+0000", 2023-11-01T04:17:08.932 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.60368778700000003 2023-11-01T04:17:08.933 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.933 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.933 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2023-11-01T04:17:08.933 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:59.447810+0000", 2023-11-01T04:17:08.934 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6200000000000001e-06 2023-11-01T04:17:08.934 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.934 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.934 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.934 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.282560+0000", 2023-11-01T04:17:08.935 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.83474988400000005 2023-11-01T04:17:08.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.935 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.935 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for blocked object", 2023-11-01T04:17:08.936 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.282663+0000", 2023-11-01T04:17:08.936 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000103696 2023-11-01T04:17:08.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.936 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.936 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.937 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.824655+0000", 2023-11-01T04:17:08.937 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.54199128200000002 2023-11-01T04:17:08.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.937 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.937 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:08.937 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.824767+0000", 2023-11-01T04:17:08.938 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011225700000000001 2023-11-01T04:17:08.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.938 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.938 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.938 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.824999+0000", 2023-11-01T04:17:08.938 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00023217100000000001 2023-11-01T04:17:08.939 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.939 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.939 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.939 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.940 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 79/31 ECSubWriteReply(tid=42, last_complete=79'24, committed=1, applied=1))", 2023-11-01T04:17:08.940 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:00.590073+0000", 2023-11-01T04:17:08.940 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 8.1283206539999995, 2023-11-01T04:17:08.940 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.200537034, 2023-11-01T04:17:08.940 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:08.940 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:08.941 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:08.941 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.941 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:08.941 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.590073+0000", 2023-11-01T04:17:08.941 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.590073+0000", 2023-11-01T04:17:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.590075+0000", 2023-11-01T04:17:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.968e-06 2023-11-01T04:17:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.590085+0000", 2023-11-01T04:17:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0677e-05 2023-11-01T04:17:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.590117+0000", 2023-11-01T04:17:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1689000000000001e-05 2023-11-01T04:17:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.590129+0000", 2023-11-01T04:17:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2285e-05 2023-11-01T04:17:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.790462+0000", 2023-11-01T04:17:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.200332916 2023-11-01T04:17:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.790610+0000", 2023-11-01T04:17:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014749899999999999 2023-11-01T04:17:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:08.949 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:08.949 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:17:08.949 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:08.950 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-11-01T04:17:08.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-01T04:17:08.963 INFO:tasks.rados.rados.0.smithi190.stdout:309: done (1 left) 2023-11-01T04:17:08.963 INFO:tasks.rados.rados.0.smithi190.stdout:312: done (0 left) 2023-11-01T04:17:08.964 INFO:tasks.rados.rados.0.smithi190.stdout:313: read oid 15 snap -1 2023-11-01T04:17:08.964 INFO:tasks.rados.rados.0.smithi190.stdout:313: expect (ObjNum 112 snap 25 seq_num 112) 2023-11-01T04:17:08.964 INFO:tasks.rados.rados.0.smithi190.stdout:314: read oid 1 snap -1 2023-11-01T04:17:08.964 INFO:tasks.rados.rados.0.smithi190.stdout:314: expect (ObjNum 0 snap 0 seq_num 0) 2023-11-01T04:17:08.964 INFO:tasks.rados.rados.0.smithi190.stdout:315: read oid 10 snap -1 2023-11-01T04:17:08.964 INFO:tasks.rados.rados.0.smithi190.stdout:315: expect (ObjNum 66 snap 6 seq_num 66) 2023-11-01T04:17:08.965 INFO:tasks.rados.rados.0.smithi190.stdout:316: copy_from oid 23 from oid 47 current snap is 29 2023-11-01T04:17:08.965 INFO:tasks.rados.rados.0.smithi190.stdout:317: read oid 5 snap -1 2023-11-01T04:17:08.965 INFO:tasks.rados.rados.0.smithi190.stdout:317: expect (ObjNum 103 snap 21 seq_num 103) 2023-11-01T04:17:08.965 INFO:tasks.rados.rados.0.smithi190.stdout:318: read oid 19 snap -1 2023-11-01T04:17:08.965 INFO:tasks.rados.rados.0.smithi190.stdout:318: expect (ObjNum 89 snap 20 seq_num 89) 2023-11-01T04:17:08.966 INFO:tasks.rados.rados.0.smithi190.stdout:319: snap_remove snap 26 2023-11-01T04:17:09.052 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:17:09.048+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:17:09.122 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:17:09.122 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:17:09.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.122 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:772 3.ds1 3:ba00e0a8:::smithi19039307-23:head [copy-from ver 2 in=55b] snapc 35=[35,34,33,2f,28,19] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2023-11-01T04:17:09.123 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:08.964422+0000", 2023-11-01T04:17:09.123 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.15812827500000001, 2023-11-01T04:17:09.123 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.15816271000000001, 2023-11-01T04:17:09.123 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.123 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:09.124 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.124 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.124 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.124 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 772 2023-11-01T04:17:09.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.125 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.125 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.125 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.125 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964422+0000", 2023-11-01T04:17:09.125 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.126 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.126 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.126 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964422+0000", 2023-11-01T04:17:09.126 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.127 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.127 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.127 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964426+0000", 2023-11-01T04:17:09.127 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.27e-06 2023-11-01T04:17:09.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.128 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.128 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.128 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964441+0000", 2023-11-01T04:17:09.128 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5193e-05 2023-11-01T04:17:09.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.129 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.129 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964480+0000", 2023-11-01T04:17:09.129 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9023000000000002e-05 2023-11-01T04:17:09.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.130 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.130 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.130 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964498+0000", 2023-11-01T04:17:09.130 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7949e-05 2023-11-01T04:17:09.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.131 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.131 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.131 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964641+0000", 2023-11-01T04:17:09.131 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014278899999999999 2023-11-01T04:17:09.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.132 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.132 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2023-11-01T04:17:09.132 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964656+0000", 2023-11-01T04:17:09.132 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5679e-05 2023-11-01T04:17:09.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.132 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.133 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.133 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.068989+0000", 2023-11-01T04:17:09.133 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104332509 2023-11-01T04:17:09.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.133 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.134 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.134 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.069114+0000", 2023-11-01T04:17:09.134 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012485999999999999 2023-11-01T04:17:09.134 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.134 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.135 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.135 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.135 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:773 3.ds1 3:ba00e0a8:::smithi19039307-23:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e88)", 2023-11-01T04:17:09.135 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:08.964540+0000", 2023-11-01T04:17:09.135 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.15801015500000001, 2023-11-01T04:17:09.136 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.15811173100000001, 2023-11-01T04:17:09.136 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.136 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "waiting for blocked object", 2023-11-01T04:17:09.136 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.136 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.137 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.137 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 773 2023-11-01T04:17:09.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.137 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.137 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.137 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.138 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964540+0000", 2023-11-01T04:17:09.138 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.138 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.138 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964540+0000", 2023-11-01T04:17:09.139 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.139 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.139 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964541+0000", 2023-11-01T04:17:09.140 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.64e-07 2023-11-01T04:17:09.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.140 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.140 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.140 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964546+0000", 2023-11-01T04:17:09.141 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7360000000000001e-06 2023-11-01T04:17:09.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.141 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.141 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.141 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964560+0000", 2023-11-01T04:17:09.141 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3816e-05 2023-11-01T04:17:09.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.142 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.142 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964573+0000", 2023-11-01T04:17:09.142 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3303e-05 2023-11-01T04:17:09.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.143 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.143 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.143 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.064837+0000", 2023-11-01T04:17:09.143 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100263669 2023-11-01T04:17:09.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.144 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.144 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:09.144 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.064846+0000", 2023-11-01T04:17:09.144 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0880000000000001e-06 2023-11-01T04:17:09.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.144 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.145 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.145 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.069335+0000", 2023-11-01T04:17:09.145 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0044892550000000002 2023-11-01T04:17:09.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.145 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.145 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for blocked object", 2023-11-01T04:17:09.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.069389+0000", 2023-11-01T04:17:09.146 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3284e-05 2023-11-01T04:17:09.146 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.146 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.146 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.147 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.147 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:777 3.as1 3:54622f20:::smithi19039307-19:head [read 0~1802240,checksum in=4b,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2023-11-01T04:17:09.147 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:08.964615+0000", 2023-11-01T04:17:09.147 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.157935357, 2023-11-01T04:17:09.147 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.158094764, 2023-11-01T04:17:09.147 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.148 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:09.148 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.148 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.148 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.148 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 777 2023-11-01T04:17:09.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.149 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.149 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.149 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.149 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964615+0000", 2023-11-01T04:17:09.149 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.150 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.150 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.150 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964615+0000", 2023-11-01T04:17:09.150 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.151 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.151 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.151 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964616+0000", 2023-11-01T04:17:09.151 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9299999999999996e-07 2023-11-01T04:17:09.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.152 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.152 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.152 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964622+0000", 2023-11-01T04:17:09.152 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2360000000000001e-06 2023-11-01T04:17:09.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.152 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.153 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.153 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964635+0000", 2023-11-01T04:17:09.153 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2867000000000001e-05 2023-11-01T04:17:09.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.153 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.153 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.154 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964644+0000", 2023-11-01T04:17:09.154 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8699999999999998e-06 2023-11-01T04:17:09.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.154 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.154 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.154 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964842+0000", 2023-11-01T04:17:09.155 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00019811000000000001 2023-11-01T04:17:09.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.155 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.155 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2023-11-01T04:17:09.155 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964860+0000", 2023-11-01T04:17:09.155 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8159999999999999e-05 2023-11-01T04:17:09.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.156 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.156 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.156 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.068724+0000", 2023-11-01T04:17:09.156 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103863784 2023-11-01T04:17:09.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.157 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.157 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.157 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.068858+0000", 2023-11-01T04:17:09.157 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013425199999999999 2023-11-01T04:17:09.157 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.157 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.158 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.158 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.158 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:778 3.as1 3.4f4462a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e88)", 2023-11-01T04:17:09.158 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:08.964668+0000", 2023-11-01T04:17:09.159 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.157882365, 2023-11-01T04:17:09.159 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.158094757, 2023-11-01T04:17:09.159 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.159 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "waiting_for_map not empty", 2023-11-01T04:17:09.159 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.159 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.160 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.160 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 778 2023-11-01T04:17:09.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.160 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.160 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.160 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.161 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964668+0000", 2023-11-01T04:17:09.161 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.161 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.161 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.161 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964668+0000", 2023-11-01T04:17:09.162 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.162 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.162 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.162 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964669+0000", 2023-11-01T04:17:09.163 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.093e-06 2023-11-01T04:17:09.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.163 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.163 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.163 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964676+0000", 2023-11-01T04:17:09.163 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7059999999999998e-06 2023-11-01T04:17:09.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.164 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.164 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.164 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964694+0000", 2023-11-01T04:17:09.164 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8450000000000001e-05 2023-11-01T04:17:09.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.165 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.165 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.165 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964709+0000", 2023-11-01T04:17:09.165 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4518999999999999e-05 2023-11-01T04:17:09.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.165 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964910+0000", 2023-11-01T04:17:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00020095700000000001 2023-11-01T04:17:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.166 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:09.167 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964921+0000", 2023-11-01T04:17:09.167 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1177e-05 2023-11-01T04:17:09.167 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.167 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.167 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.168 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.168 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:779 3.as1 3.4f4462a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e88)", 2023-11-01T04:17:09.168 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:08.964732+0000", 2023-11-01T04:17:09.168 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.15781889700000001, 2023-11-01T04:17:09.168 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.158085538, 2023-11-01T04:17:09.168 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.169 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "waiting_for_map not empty", 2023-11-01T04:17:09.169 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.169 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.169 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.169 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 779 2023-11-01T04:17:09.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.170 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.170 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.170 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.170 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964732+0000", 2023-11-01T04:17:09.170 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.171 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.171 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.171 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964732+0000", 2023-11-01T04:17:09.171 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.172 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.172 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.172 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964733+0000", 2023-11-01T04:17:09.172 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.063e-06 2023-11-01T04:17:09.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.172 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.173 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.173 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964740+0000", 2023-11-01T04:17:09.173 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5329999999999997e-06 2023-11-01T04:17:09.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.173 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.173 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.174 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964758+0000", 2023-11-01T04:17:09.174 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8128999999999999e-05 2023-11-01T04:17:09.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.174 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.174 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.175 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964784+0000", 2023-11-01T04:17:09.175 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5296e-05 2023-11-01T04:17:09.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.175 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.175 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.175 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.064844+0000", 2023-11-01T04:17:09.176 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100060152 2023-11-01T04:17:09.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.176 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.176 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:09.176 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.064854+0000", 2023-11-01T04:17:09.177 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.012e-05 2023-11-01T04:17:09.177 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.177 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.177 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.177 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.178 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(osd.4.22:3 3.as1 3.d75f98fa (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e88)", 2023-11-01T04:17:09.178 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:09.069305+0000", 2023-11-01T04:17:09.178 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.053246069, 2023-11-01T04:17:09.178 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.053554670999999998, 2023-11-01T04:17:09.178 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.179 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "queued for pg", 2023-11-01T04:17:09.179 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.179 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.179 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.179 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.069305+0000", 2023-11-01T04:17:09.179 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.180 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.180 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.180 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.069305+0000", 2023-11-01T04:17:09.180 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.181 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.181 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.181 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.069306+0000", 2023-11-01T04:17:09.181 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7889999999999999e-06 2023-11-01T04:17:09.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.182 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.182 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.182 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.069314+0000", 2023-11-01T04:17:09.182 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1019999999999997e-06 2023-11-01T04:17:09.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.182 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.183 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.183 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.069330+0000", 2023-11-01T04:17:09.183 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5953000000000001e-05 2023-11-01T04:17:09.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.183 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.184 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.184 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.069341+0000", 2023-11-01T04:17:09.184 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0614999999999999e-05 2023-11-01T04:17:09.184 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.184 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.184 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.185 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.185 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpRead(3.as1 88/68 ECSubRead(tid=39, to_read={3:54622f20:::smithi19039307-19:head=0,450560,0}, subchunks={3:54622f20:::smithi19039307-19:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:17:09.185 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:09.069598+0000", 2023-11-01T04:17:09.185 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.052953010000000002, 2023-11-01T04:17:09.185 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.053297283000000001, 2023-11-01T04:17:09.186 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.186 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "queued for pg", 2023-11-01T04:17:09.186 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.186 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.186 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.186 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.069598+0000", 2023-11-01T04:17:09.187 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.187 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.187 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.187 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.069598+0000", 2023-11-01T04:17:09.187 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.188 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.188 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.188 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.069598+0000", 2023-11-01T04:17:09.188 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.189 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.189 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.069598+0000", 2023-11-01T04:17:09.189 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.190 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.190 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.069620+0000", 2023-11-01T04:17:09.190 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1906e-05 2023-11-01T04:17:09.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.190 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.191 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.191 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.069629+0000", 2023-11-01T04:17:09.191 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.9720000000000004e-06 2023-11-01T04:17:09.191 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.191 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.191 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.192 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.as1 87/68 ECSubReadReply(tid=39, attrs_read=0))", 2023-11-01T04:17:09.192 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:09.070816+0000", 2023-11-01T04:17:09.192 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.051735061999999998, 2023-11-01T04:17:09.192 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.052110554000000003, 2023-11-01T04:17:09.193 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.193 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "no flag points reached", 2023-11-01T04:17:09.193 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.193 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.193 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.070816+0000", 2023-11-01T04:17:09.194 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.194 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.194 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.194 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.070816+0000", 2023-11-01T04:17:09.194 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.195 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.195 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.195 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.070817+0000", 2023-11-01T04:17:09.195 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1689999999999999e-06 2023-11-01T04:17:09.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.196 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.196 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.196 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.071831+0000", 2023-11-01T04:17:09.196 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0010146070000000001 2023-11-01T04:17:09.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.196 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.197 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.197 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.071851+0000", 2023-11-01T04:17:09.197 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9928999999999998e-05 2023-11-01T04:17:09.197 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.197 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.197 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.198 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.198 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpReadReply(3.as1 88/68 ECSubReadReply(tid=39, attrs_read=0))", 2023-11-01T04:17:09.198 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:09.071447+0000", 2023-11-01T04:17:09.198 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.051103408000000003, 2023-11-01T04:17:09.198 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.051505197000000003, 2023-11-01T04:17:09.199 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.199 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "queued for pg", 2023-11-01T04:17:09.199 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.199 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.199 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.200 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.071447+0000", 2023-11-01T04:17:09.200 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.200 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.200 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.200 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.071447+0000", 2023-11-01T04:17:09.201 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.201 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.201 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.201 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.071448+0000", 2023-11-01T04:17:09.202 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.7700000000000003e-07 2023-11-01T04:17:09.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.202 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.202 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.202 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.072323+0000", 2023-11-01T04:17:09.202 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00087537800000000001 2023-11-01T04:17:09.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.203 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.203 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.203 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.072338+0000", 2023-11-01T04:17:09.203 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4768e-05 2023-11-01T04:17:09.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.204 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.204 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.204 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.072346+0000", 2023-11-01T04:17:09.204 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5669999999999997e-06 2023-11-01T04:17:09.204 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.205 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.205 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.205 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.205 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2023-11-01T04:17:09.205 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 9 2023-11-01T04:17:09.205 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:17:09.207 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-01T04:17:09.207 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:09.204+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:09.308 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:17:09.309+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:17:09.325 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:17:09.325 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:17:09.325 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2023-11-01T04:17:09.326 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2023-11-01T04:17:09.326 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:17:09.339 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-01T04:17:09.408 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:17:09.407+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:17:09.458 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:17:09.459 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2023-11-01T04:17:09.459 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2023-11-01T04:17:09.459 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:17:09.459 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.459 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:536 3.as1 3:54622f20:::smithi19039307-19:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:09.460 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:47.776853+0000", 2023-11-01T04:17:09.460 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 21.679165511000001, 2023-11-01T04:17:09.460 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6173196650000001, 2023-11-01T04:17:09.460 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.460 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:09.461 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.461 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.461 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.461 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 536 2023-11-01T04:17:09.461 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.462 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.462 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.462 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.462 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.776853+0000", 2023-11-01T04:17:09.462 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.463 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.463 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.463 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.463 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.776853+0000", 2023-11-01T04:17:09.463 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.463 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.464 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.464 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.464 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.776855+0000", 2023-11-01T04:17:09.464 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.463e-06 2023-11-01T04:17:09.464 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.465 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.465 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.465 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.776864+0000", 2023-11-01T04:17:09.465 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5529999999999993e-06 2023-11-01T04:17:09.465 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.466 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.466 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.466 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.776886+0000", 2023-11-01T04:17:09.466 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2447e-05 2023-11-01T04:17:09.466 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.467 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.467 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.467 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.776903+0000", 2023-11-01T04:17:09.467 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6609e-05 2023-11-01T04:17:09.467 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.468 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.468 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.468 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.777058+0000", 2023-11-01T04:17:09.468 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015479799999999999 2023-11-01T04:17:09.468 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.468 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.469 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2023-11-01T04:17:09.469 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.777071+0000", 2023-11-01T04:17:09.469 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2965999999999999e-05 2023-11-01T04:17:09.469 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.469 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.469 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.470 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.187223+0000", 2023-11-01T04:17:09.470 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.41015271599999997 2023-11-01T04:17:09.470 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.470 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.470 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2023-11-01T04:17:09.471 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.187230+0000", 2023-11-01T04:17:09.471 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.046e-06 2023-11-01T04:17:09.471 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.471 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.471 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.472 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.191371+0000", 2023-11-01T04:17:09.472 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.004140748 2023-11-01T04:17:09.472 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.472 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.472 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.472 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.191657+0000", 2023-11-01T04:17:09.473 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00028538899999999999 2023-11-01T04:17:09.473 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.473 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.473 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.473 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:49.394172+0000", 2023-11-01T04:17:09.473 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20251593000000001 2023-11-01T04:17:09.474 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.474 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.474 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.474 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.474 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.475 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s3 68/31 ECSubWrite(tid=315, reqid=osd.13.0:315, at_version=68'72, trim_to=0'0, roll_forward_to=67'68))", 2023-11-01T04:17:09.475 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:47.788107+0000", 2023-11-01T04:17:09.475 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 21.667911171, 2023-11-01T04:17:09.475 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.39927050600000003, 2023-11-01T04:17:09.475 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.475 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:09.476 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.476 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.476 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.476 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.788107+0000", 2023-11-01T04:17:09.476 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.477 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.477 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.477 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.477 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.788107+0000", 2023-11-01T04:17:09.477 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.477 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.478 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.478 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.478 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.788108+0000", 2023-11-01T04:17:09.478 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0699999999999997e-07 2023-11-01T04:17:09.478 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.479 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.479 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.479 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.788146+0000", 2023-11-01T04:17:09.479 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8209e-05 2023-11-01T04:17:09.479 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.479 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.480 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.480 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.788163+0000", 2023-11-01T04:17:09.480 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6900000000000001e-05 2023-11-01T04:17:09.480 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.480 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.481 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.481 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.788176+0000", 2023-11-01T04:17:09.481 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.272e-05 2023-11-01T04:17:09.481 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.481 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.481 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.482 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.086965+0000", 2023-11-01T04:17:09.482 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.29878976600000001 2023-11-01T04:17:09.482 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.482 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.482 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:09.483 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.086990+0000", 2023-11-01T04:17:09.483 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4457e-05 2023-11-01T04:17:09.483 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.483 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.483 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:09.484 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.187306+0000", 2023-11-01T04:17:09.484 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100315998 2023-11-01T04:17:09.484 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.484 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.484 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.484 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.187378+0000", 2023-11-01T04:17:09.485 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1848999999999996e-05 2023-11-01T04:17:09.485 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.485 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.485 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.485 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.486 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.486 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:619 3.ds1 3:ba00e0a8:::smithi19039307-23:head [append 0~589824] snapc 27=[27,26,22,1b,19,18,12,11] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2023-11-01T04:17:09.486 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:54.774547+0000", 2023-11-01T04:17:09.486 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 14.681471605, 2023-11-01T04:17:09.486 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.32359992700000001, 2023-11-01T04:17:09.487 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.487 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:09.487 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.487 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.487 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.487 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 619 2023-11-01T04:17:09.488 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.488 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.488 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.488 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.488 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.774547+0000", 2023-11-01T04:17:09.488 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.489 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.489 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.489 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.489 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.774547+0000", 2023-11-01T04:17:09.489 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.490 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.490 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.490 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.490 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.774550+0000", 2023-11-01T04:17:09.490 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8399999999999999e-06 2023-11-01T04:17:09.490 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.491 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.491 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.491 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.775642+0000", 2023-11-01T04:17:09.491 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0010924999999999999 2023-11-01T04:17:09.491 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.492 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.492 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.492 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.775670+0000", 2023-11-01T04:17:09.492 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7517000000000001e-05 2023-11-01T04:17:09.492 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.492 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.493 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.493 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.775684+0000", 2023-11-01T04:17:09.493 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4112e-05 2023-11-01T04:17:09.493 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.493 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.493 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.494 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.782384+0000", 2023-11-01T04:17:09.494 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0067004439999999998 2023-11-01T04:17:09.494 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.494 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.494 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.494 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.782503+0000", 2023-11-01T04:17:09.495 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000118773 2023-11-01T04:17:09.495 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.495 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.495 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:09.495 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.783538+0000", 2023-11-01T04:17:09.495 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001035338 2023-11-01T04:17:09.496 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.496 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.496 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:09.496 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.785653+0000", 2023-11-01T04:17:09.496 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002114997 2023-11-01T04:17:09.497 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.497 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.497 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:09.497 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.097825+0000", 2023-11-01T04:17:09.497 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.31217141999999998 2023-11-01T04:17:09.497 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.498 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.498 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.498 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.098147+0000", 2023-11-01T04:17:09.498 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00032198600000000002 2023-11-01T04:17:09.498 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.499 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.499 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.499 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.499 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.499 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:620 3.ds1 3:ba00e0a8:::smithi19039307-23:head [append 0~540672] snapc 27=[27,26,22,1b,19,18,12,11] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2023-11-01T04:17:09.499 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:54.781704+0000", 2023-11-01T04:17:09.500 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 14.674314756999999, 2023-11-01T04:17:09.500 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.31728704400000002, 2023-11-01T04:17:09.500 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.500 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:09.500 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.501 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.501 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.501 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 620 2023-11-01T04:17:09.501 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.501 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.502 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.502 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.502 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.781704+0000", 2023-11-01T04:17:09.502 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.502 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.503 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.503 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.503 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.781704+0000", 2023-11-01T04:17:09.503 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.503 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.503 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.504 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.504 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.781706+0000", 2023-11-01T04:17:09.504 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4549999999999998e-06 2023-11-01T04:17:09.504 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.504 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.504 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.505 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.782222+0000", 2023-11-01T04:17:09.505 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00051576599999999997 2023-11-01T04:17:09.505 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.505 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.505 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.506 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.782250+0000", 2023-11-01T04:17:09.506 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7750999999999999e-05 2023-11-01T04:17:09.506 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.506 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.506 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.506 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.782267+0000", 2023-11-01T04:17:09.507 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7716000000000001e-05 2023-11-01T04:17:09.507 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.507 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.507 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.507 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.785897+0000", 2023-11-01T04:17:09.507 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0036297730000000002 2023-11-01T04:17:09.508 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.508 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.508 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.508 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.786001+0000", 2023-11-01T04:17:09.508 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010424099999999999 2023-11-01T04:17:09.508 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.509 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.509 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:09.509 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.786754+0000", 2023-11-01T04:17:09.509 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000753091 2023-11-01T04:17:09.509 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.509 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.510 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:09.510 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.788391+0000", 2023-11-01T04:17:09.510 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001636289 2023-11-01T04:17:09.510 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.510 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.511 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:09.511 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.098663+0000", 2023-11-01T04:17:09.511 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.31027191900000001 2023-11-01T04:17:09.511 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.511 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.511 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.512 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.098991+0000", 2023-11-01T04:17:09.512 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00032804299999999998 2023-11-01T04:17:09.512 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.512 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.512 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.512 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.513 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.513 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:621 3.ds1 3:ba00e0a8:::smithi19039307-23:head [append 0~622592] snapc 27=[27,26,22,1b,19,18,12,11] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2023-11-01T04:17:09.513 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:54.789814+0000", 2023-11-01T04:17:09.513 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 14.666204663, 2023-11-01T04:17:09.513 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30989988200000002, 2023-11-01T04:17:09.514 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.514 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:09.514 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.514 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.514 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.514 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 621 2023-11-01T04:17:09.515 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.515 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.515 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.515 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.515 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.789814+0000", 2023-11-01T04:17:09.516 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.516 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.516 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.516 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.516 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.789814+0000", 2023-11-01T04:17:09.516 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.517 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.517 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.517 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.517 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.789815+0000", 2023-11-01T04:17:09.517 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.556e-06 2023-11-01T04:17:09.518 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.518 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.518 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.518 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790468+0000", 2023-11-01T04:17:09.518 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000652453 2023-11-01T04:17:09.518 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.519 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.519 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.519 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790486+0000", 2023-11-01T04:17:09.519 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7861000000000001e-05 2023-11-01T04:17:09.519 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.519 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.520 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.520 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790494+0000", 2023-11-01T04:17:09.520 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5450000000000003e-06 2023-11-01T04:17:09.520 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.520 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.520 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.521 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.890901+0000", 2023-11-01T04:17:09.521 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100406963 2023-11-01T04:17:09.521 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.521 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.521 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.522 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.891217+0000", 2023-11-01T04:17:09.522 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00031643900000000001 2023-11-01T04:17:09.522 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.522 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.522 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:09.522 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.893720+0000", 2023-11-01T04:17:09.523 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0025027389999999999 2023-11-01T04:17:09.523 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.523 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.523 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:09.523 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.896858+0000", 2023-11-01T04:17:09.523 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0031375919999999998 2023-11-01T04:17:09.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.524 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.524 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:09.524 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.099333+0000", 2023-11-01T04:17:09.524 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20247476 2023-11-01T04:17:09.524 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.525 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.525 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.525 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.099714+0000", 2023-11-01T04:17:09.525 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00038097400000000002 2023-11-01T04:17:09.525 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.526 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.526 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.526 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.526 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.526 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:622 3.ds1 3:ba00e0a8:::smithi19039307-23:head [setxattr _header (57) in=7b] snapc 27=[27,26,22,1b,19,18,12,11] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2023-11-01T04:17:09.527 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:54.790514+0000", 2023-11-01T04:17:09.527 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 14.665504305000001, 2023-11-01T04:17:09.527 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.51671497700000002, 2023-11-01T04:17:09.527 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.527 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:09.528 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.528 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.528 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.528 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 622 2023-11-01T04:17:09.528 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.528 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.529 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.529 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.529 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790514+0000", 2023-11-01T04:17:09.529 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.529 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.530 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.530 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.530 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790514+0000", 2023-11-01T04:17:09.530 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.530 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.531 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.531 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.531 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790515+0000", 2023-11-01T04:17:09.531 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.133e-06 2023-11-01T04:17:09.531 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.531 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.532 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.532 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790518+0000", 2023-11-01T04:17:09.532 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.123e-06 2023-11-01T04:17:09.532 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.532 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.532 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.533 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790527+0000", 2023-11-01T04:17:09.533 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8449999999999992e-06 2023-11-01T04:17:09.533 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.533 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.533 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.534 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790534+0000", 2023-11-01T04:17:09.534 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8739999999999997e-06 2023-11-01T04:17:09.534 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.534 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.534 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.535 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.100566+0000", 2023-11-01T04:17:09.535 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.31003165700000002 2023-11-01T04:17:09.535 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.535 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.535 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.536 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.100855+0000", 2023-11-01T04:17:09.536 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000289074 2023-11-01T04:17:09.536 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.536 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.536 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:09.537 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.102192+0000", 2023-11-01T04:17:09.537 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001336814 2023-11-01T04:17:09.537 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.537 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.537 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:09.538 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.104907+0000", 2023-11-01T04:17:09.538 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0027159020000000001 2023-11-01T04:17:09.538 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.538 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.538 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:09.539 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.306899+0000", 2023-11-01T04:17:09.539 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20199112599999999 2023-11-01T04:17:09.539 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.539 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.539 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.540 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.307229+0000", 2023-11-01T04:17:09.540 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00033042899999999998 2023-11-01T04:17:09.540 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.540 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.540 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.540 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.541 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.541 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:623 3.ds1 3:ba00e0a8:::smithi19039307-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2023-11-01T04:17:09.541 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:54.790541+0000", 2023-11-01T04:17:09.541 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 14.665477337, 2023-11-01T04:17:09.541 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.71895732199999995, 2023-11-01T04:17:09.542 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.542 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:09.542 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.542 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.542 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.543 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 623 2023-11-01T04:17:09.543 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.543 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.543 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.543 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.543 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790541+0000", 2023-11-01T04:17:09.544 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.544 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.544 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.544 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.544 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790541+0000", 2023-11-01T04:17:09.544 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.545 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.545 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.545 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.545 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790541+0000", 2023-11-01T04:17:09.545 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.58e-07 2023-11-01T04:17:09.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.546 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.546 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.546 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790544+0000", 2023-11-01T04:17:09.546 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8420000000000001e-06 2023-11-01T04:17:09.546 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.547 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.547 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.547 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790552+0000", 2023-11-01T04:17:09.547 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4789999999999999e-06 2023-11-01T04:17:09.547 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.548 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.548 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.548 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:54.790556+0000", 2023-11-01T04:17:09.548 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5920000000000002e-06 2023-11-01T04:17:09.548 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.548 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.102797+0000", 2023-11-01T04:17:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.31224017199999998 2023-11-01T04:17:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.549 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.102961+0000", 2023-11-01T04:17:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016474999999999999 2023-11-01T04:17:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.550 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.407367+0000", 2023-11-01T04:17:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30440549900000002 2023-11-01T04:17:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.551 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.407516+0000", 2023-11-01T04:17:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014955400000000001 2023-11-01T04:17:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.552 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:55.509498+0000", 2023-11-01T04:17:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.101981976 2023-11-01T04:17:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.553 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:669 3.ds1 3:b0d0a55f:::smithi19039307-8:head [append 0~655360] snapc 28=[28,26,22,19,18] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2023-11-01T04:17:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:00.195869+0000", 2023-11-01T04:17:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 9.26014947, 2023-11-01T04:17:09.554 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.32182761300000001, 2023-11-01T04:17:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.555 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 669 2023-11-01T04:17:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.556 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.195869+0000", 2023-11-01T04:17:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.557 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.195869+0000", 2023-11-01T04:17:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.558 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.195871+0000", 2023-11-01T04:17:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7630000000000001e-06 2023-11-01T04:17:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.196752+0000", 2023-11-01T04:17:09.559 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00088137199999999995 2023-11-01T04:17:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.196772+0000", 2023-11-01T04:17:09.560 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0392000000000001e-05 2023-11-01T04:17:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.196786+0000", 2023-11-01T04:17:09.561 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3888e-05 2023-11-01T04:17:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.297010+0000", 2023-11-01T04:17:09.562 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10022334300000001 2023-11-01T04:17:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.297200+0000", 2023-11-01T04:17:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000190349 2023-11-01T04:17:09.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.299230+0000", 2023-11-01T04:17:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002030025 2023-11-01T04:17:09.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.412760+0000", 2023-11-01T04:17:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.113529531 2023-11-01T04:17:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.517304+0000", 2023-11-01T04:17:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104544391 2023-11-01T04:17:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.566 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.517696+0000", 2023-11-01T04:17:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00039255900000000001 2023-11-01T04:17:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.567 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.568 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.568 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:670 3.ds1 3:b0d0a55f:::smithi19039307-8:head [append 0~606208] snapc 28=[28,26,22,19,18] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2023-11-01T04:17:09.568 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:00.205650+0000", 2023-11-01T04:17:09.568 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 9.2503686750000007, 2023-11-01T04:17:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.31266979099999997, 2023-11-01T04:17:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.569 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 670 2023-11-01T04:17:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.570 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.205650+0000", 2023-11-01T04:17:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.571 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.205650+0000", 2023-11-01T04:17:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.572 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.205651+0000", 2023-11-01T04:17:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6249999999999999e-06 2023-11-01T04:17:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.206187+0000", 2023-11-01T04:17:09.573 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00053543199999999997 2023-11-01T04:17:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.206204+0000", 2023-11-01T04:17:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7659000000000002e-05 2023-11-01T04:17:09.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.206213+0000", 2023-11-01T04:17:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5909999999999996e-06 2023-11-01T04:17:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.575 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.301951+0000", 2023-11-01T04:17:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.095738555000000003 2023-11-01T04:17:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.576 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.302131+0000", 2023-11-01T04:17:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00017976999999999999 2023-11-01T04:17:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.303576+0000", 2023-11-01T04:17:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0014449630000000001 2023-11-01T04:17:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.578 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.412765+0000", 2023-11-01T04:17:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.109188699 2023-11-01T04:17:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.579 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.518018+0000", 2023-11-01T04:17:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.105253238 2023-11-01T04:17:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.580 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.518319+0000", 2023-11-01T04:17:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00030125900000000001 2023-11-01T04:17:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.581 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:672 3.ds1 3:b0d0a55f:::smithi19039307-8:head [append 0~81920] snapc 28=[28,26,22,19,18] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2023-11-01T04:17:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:00.214469+0000", 2023-11-01T04:17:09.582 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 9.2415495179999994, 2023-11-01T04:17:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.60555795599999995, 2023-11-01T04:17:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.583 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 672 2023-11-01T04:17:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214469+0000", 2023-11-01T04:17:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214469+0000", 2023-11-01T04:17:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.586 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214470+0000", 2023-11-01T04:17:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.879999999999999e-07 2023-11-01T04:17:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214573+0000", 2023-11-01T04:17:09.587 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000103336 2023-11-01T04:17:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214582+0000", 2023-11-01T04:17:09.588 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4509999999999992e-06 2023-11-01T04:17:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214586+0000", 2023-11-01T04:17:09.589 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4870000000000004e-06 2023-11-01T04:17:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.309040+0000", 2023-11-01T04:17:09.590 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.094454366999999997 2023-11-01T04:17:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.309379+0000", 2023-11-01T04:17:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00033828800000000001 2023-11-01T04:17:09.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.592 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.592 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:09.592 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.311051+0000", 2023-11-01T04:17:09.592 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001672434 2023-11-01T04:17:09.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.593 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.593 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:09.593 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.412915+0000", 2023-11-01T04:17:09.593 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10186337400000001 2023-11-01T04:17:09.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.593 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.594 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:09.594 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.819673+0000", 2023-11-01T04:17:09.594 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.40675862800000001 2023-11-01T04:17:09.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.594 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.595 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.595 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.820027+0000", 2023-11-01T04:17:09.595 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00035360300000000002 2023-11-01T04:17:09.595 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.595 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.595 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.596 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.596 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.596 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:673 3.ds1 3:b0d0a55f:::smithi19039307-8:head [setxattr _header (56) in=7b] snapc 28=[28,26,22,19,18] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2023-11-01T04:17:09.596 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:00.214594+0000", 2023-11-01T04:17:09.596 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 9.2414244700000001, 2023-11-01T04:17:09.597 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.60596670500000005, 2023-11-01T04:17:09.597 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.597 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:09.597 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.597 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.598 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.598 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 673 2023-11-01T04:17:09.598 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.598 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.598 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.599 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.599 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214594+0000", 2023-11-01T04:17:09.599 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.599 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.600 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.600 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214594+0000", 2023-11-01T04:17:09.600 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.600 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.600 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.601 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214594+0000", 2023-11-01T04:17:09.601 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5100000000000001e-07 2023-11-01T04:17:09.601 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.601 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.601 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.601 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214597+0000", 2023-11-01T04:17:09.602 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4260000000000002e-06 2023-11-01T04:17:09.602 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.602 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.602 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.602 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214601+0000", 2023-11-01T04:17:09.603 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8859999999999997e-06 2023-11-01T04:17:09.603 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.603 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.603 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.603 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214604+0000", 2023-11-01T04:17:09.603 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0290000000000001e-06 2023-11-01T04:17:09.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.604 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.604 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.604 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.413491+0000", 2023-11-01T04:17:09.604 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19888792299999999 2023-11-01T04:17:09.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.605 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.605 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.605 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.413679+0000", 2023-11-01T04:17:09.605 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000187226 2023-11-01T04:17:09.605 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.605 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.606 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:09.606 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.414429+0000", 2023-11-01T04:17:09.606 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00075007899999999996 2023-11-01T04:17:09.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.607 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.607 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:09.607 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.516993+0000", 2023-11-01T04:17:09.607 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.102563949 2023-11-01T04:17:09.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.607 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.608 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:09.608 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.820225+0000", 2023-11-01T04:17:09.608 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30323224500000001 2023-11-01T04:17:09.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.608 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.608 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.609 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.820561+0000", 2023-11-01T04:17:09.609 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00033559099999999999 2023-11-01T04:17:09.609 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.609 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.609 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.610 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.610 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:674 3.ds1 3:b0d0a55f:::smithi19039307-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2023-11-01T04:17:09.610 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:00.214614+0000", 2023-11-01T04:17:09.610 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 9.241404503, 2023-11-01T04:17:09.611 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.90944939499999999, 2023-11-01T04:17:09.611 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.611 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:09.611 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.611 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.612 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.612 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 674 2023-11-01T04:17:09.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.612 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.612 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.612 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.613 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214614+0000", 2023-11-01T04:17:09.613 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.613 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.613 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.613 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214614+0000", 2023-11-01T04:17:09.614 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.614 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.614 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.614 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.614 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214614+0000", 2023-11-01T04:17:09.614 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.46e-07 2023-11-01T04:17:09.615 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.615 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.615 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.615 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214616+0000", 2023-11-01T04:17:09.615 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8029999999999999e-06 2023-11-01T04:17:09.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.616 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.616 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.616 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.214621+0000", 2023-11-01T04:17:09.616 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.729e-06 2023-11-01T04:17:09.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.617 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.617 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.617 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.314694+0000", 2023-11-01T04:17:09.617 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100073327 2023-11-01T04:17:09.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.617 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.618 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.618 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.416440+0000", 2023-11-01T04:17:09.618 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10174552100000001 2023-11-01T04:17:09.618 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.618 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.618 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:09.619 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.416736+0000", 2023-11-01T04:17:09.619 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00029640800000000001 2023-11-01T04:17:09.619 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.619 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.619 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.619 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.821521+0000", 2023-11-01T04:17:09.620 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.40478484199999998 2023-11-01T04:17:09.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.620 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.620 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.620 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.821589+0000", 2023-11-01T04:17:09.620 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8559999999999994e-05 2023-11-01T04:17:09.621 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.621 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.621 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.621 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:01.124063+0000", 2023-11-01T04:17:09.621 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30247385900000001 2023-11-01T04:17:09.622 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.622 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.622 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.622 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.622 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.623 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(osd.0.22:1 3.ds1 3:b31c2ca6:::smithi19039307-48:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2023-11-01T04:17:09.623 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:00.315971+0000", 2023-11-01T04:17:09.623 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 9.1400472570000009, 2023-11-01T04:17:09.623 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.50617702600000003, 2023-11-01T04:17:09.623 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.623 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:09.624 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.624 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.624 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.624 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.315971+0000", 2023-11-01T04:17:09.624 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.625 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.625 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.625 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.315971+0000", 2023-11-01T04:17:09.625 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.626 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.626 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.626 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.315973+0000", 2023-11-01T04:17:09.626 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.477e-06 2023-11-01T04:17:09.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.626 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.627 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.627 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.315979+0000", 2023-11-01T04:17:09.627 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.1360000000000002e-06 2023-11-01T04:17:09.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.627 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.628 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.628 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.315992+0000", 2023-11-01T04:17:09.628 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3628000000000001e-05 2023-11-01T04:17:09.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.628 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.628 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.629 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.316001+0000", 2023-11-01T04:17:09.629 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5490000000000006e-06 2023-11-01T04:17:09.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.629 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.629 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.630 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.415117+0000", 2023-11-01T04:17:09.630 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.099116335 2023-11-01T04:17:09.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.630 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.631 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.631 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.415426+0000", 2023-11-01T04:17:09.631 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000308585 2023-11-01T04:17:09.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.631 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.631 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.632 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:00.822148+0000", 2023-11-01T04:17:09.632 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.40672231599999997 2023-11-01T04:17:09.632 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.632 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.632 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.633 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.633 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:698 3.ds1 3:b53d1bb4:::smithi19039307-22:head [read 0~360448,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:09.633 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:03.411595+0000", 2023-11-01T04:17:09.633 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.0444228549999997, 2023-11-01T04:17:09.633 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.90948891200000004, 2023-11-01T04:17:09.634 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.634 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:09.634 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.634 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.634 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.634 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 698 2023-11-01T04:17:09.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.635 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.635 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.635 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.411595+0000", 2023-11-01T04:17:09.636 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.636 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.636 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.411595+0000", 2023-11-01T04:17:09.636 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.637 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.637 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.411597+0000", 2023-11-01T04:17:09.637 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9400000000000001e-06 2023-11-01T04:17:09.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.638 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.638 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.411605+0000", 2023-11-01T04:17:09.638 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9959999999999993e-06 2023-11-01T04:17:09.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.639 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.639 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.411616+0000", 2023-11-01T04:17:09.639 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1106e-05 2023-11-01T04:17:09.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.640 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.640 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.511761+0000", 2023-11-01T04:17:09.640 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100144235 2023-11-01T04:17:09.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.640 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.640 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.641 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.511946+0000", 2023-11-01T04:17:09.641 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00018549699999999999 2023-11-01T04:17:09.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.641 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.641 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.512229+0000", 2023-11-01T04:17:09.642 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00028272099999999999 2023-11-01T04:17:09.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.642 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.642 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.642 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.321084+0000", 2023-11-01T04:17:09.643 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.80885541699999997 2023-11-01T04:17:09.643 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.643 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.643 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.643 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.644 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:699 3.ds1 3:b53d1bb4:::smithi19039307-22:head [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:09.644 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:03.511810+0000", 2023-11-01T04:17:09.644 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.9442079889999997, 2023-11-01T04:17:09.644 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.81061567300000004, 2023-11-01T04:17:09.644 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.644 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:09.645 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.645 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.645 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.645 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 699 2023-11-01T04:17:09.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.645 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.646 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.646 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.646 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.511810+0000", 2023-11-01T04:17:09.646 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.646 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.647 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.647 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.511810+0000", 2023-11-01T04:17:09.647 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.647 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.647 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.511812+0000", 2023-11-01T04:17:09.648 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1430000000000001e-06 2023-11-01T04:17:09.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.648 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.511826+0000", 2023-11-01T04:17:09.649 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3098999999999999e-05 2023-11-01T04:17:09.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.649 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.649 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.649 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.511856+0000", 2023-11-01T04:17:09.649 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0088999999999999e-05 2023-11-01T04:17:09.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.650 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.650 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.650 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612020+0000", 2023-11-01T04:17:09.650 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10016425800000001 2023-11-01T04:17:09.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.651 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.651 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.651 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.613967+0000", 2023-11-01T04:17:09.651 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001947334 2023-11-01T04:17:09.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.652 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.652 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.652 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.614286+0000", 2023-11-01T04:17:09.652 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00031903900000000002 2023-11-01T04:17:09.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.652 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.653 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.653 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.322426+0000", 2023-11-01T04:17:09.653 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.70813971099999995 2023-11-01T04:17:09.653 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.653 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.653 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.654 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.654 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:700 3.ds1 3:b53d1bb4:::smithi19039307-22:head [read 0~360448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:09.654 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:03.612054+0000", 2023-11-01T04:17:09.654 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.8439642379999999, 2023-11-01T04:17:09.654 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.71148887800000005, 2023-11-01T04:17:09.655 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.655 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:09.655 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.655 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.655 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.655 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 700 2023-11-01T04:17:09.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.656 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.656 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.656 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.656 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612054+0000", 2023-11-01T04:17:09.656 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.657 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.657 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.657 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612054+0000", 2023-11-01T04:17:09.657 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.658 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.658 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.658 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612056+0000", 2023-11-01T04:17:09.658 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.753e-06 2023-11-01T04:17:09.658 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.659 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.659 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.659 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612068+0000", 2023-11-01T04:17:09.659 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2244e-05 2023-11-01T04:17:09.659 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.660 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.660 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.660 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612087+0000", 2023-11-01T04:17:09.660 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9045000000000001e-05 2023-11-01T04:17:09.660 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.660 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.661 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.661 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612103+0000", 2023-11-01T04:17:09.661 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6169999999999999e-05 2023-11-01T04:17:09.661 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.661 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.662 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.662 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.615266+0000", 2023-11-01T04:17:09.662 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0031631150000000002 2023-11-01T04:17:09.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.662 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.662 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.663 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.615551+0000", 2023-11-01T04:17:09.663 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00028439300000000003 2023-11-01T04:17:09.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.663 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.663 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.323543+0000", 2023-11-01T04:17:09.664 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.70799215800000004 2023-11-01T04:17:09.664 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.664 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.664 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.664 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.665 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:722 3.ds1 3:b0d0a55f:::smithi19039307-8:head [append 0~0] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:09.665 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:03.612131+0000", 2023-11-01T04:17:09.665 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.8438874639999998, 2023-11-01T04:17:09.665 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.70728016800000004, 2023-11-01T04:17:09.665 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.666 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:09.666 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.666 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.666 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.666 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 722 2023-11-01T04:17:09.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.667 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.667 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.667 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.667 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612131+0000", 2023-11-01T04:17:09.667 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.668 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.668 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.668 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612131+0000", 2023-11-01T04:17:09.668 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.669 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.669 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.669 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612132+0000", 2023-11-01T04:17:09.669 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.08e-06 2023-11-01T04:17:09.669 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.670 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.670 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.670 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612136+0000", 2023-11-01T04:17:09.670 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1799999999999998e-06 2023-11-01T04:17:09.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.670 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.671 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.671 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612151+0000", 2023-11-01T04:17:09.671 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5342999999999999e-05 2023-11-01T04:17:09.671 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.671 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.672 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.672 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612160+0000", 2023-11-01T04:17:09.672 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9930000000000001e-06 2023-11-01T04:17:09.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.672 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.673 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.712501+0000", 2023-11-01T04:17:09.673 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100340858 2023-11-01T04:17:09.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.673 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.673 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.712687+0000", 2023-11-01T04:17:09.674 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000186083 2023-11-01T04:17:09.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.674 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:09.674 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.714029+0000", 2023-11-01T04:17:09.675 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0013412579999999999 2023-11-01T04:17:09.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.675 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:09.675 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.720658+0000", 2023-11-01T04:17:09.675 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0066293869999999996 2023-11-01T04:17:09.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.676 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:09.676 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.319057+0000", 2023-11-01T04:17:09.676 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.59839941200000002 2023-11-01T04:17:09.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.677 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.677 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.677 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.319411+0000", 2023-11-01T04:17:09.677 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000353574 2023-11-01T04:17:09.678 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.678 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.678 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.678 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.678 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.678 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:723 3.ds1 3:b0d0a55f:::smithi19039307-8:head [rollback 2f] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:09.679 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:03.612172+0000", 2023-11-01T04:17:09.679 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.8438465150000001, 2023-11-01T04:17:09.679 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.81017731900000001, 2023-11-01T04:17:09.679 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.679 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:09.680 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.680 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.680 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.680 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 723 2023-11-01T04:17:09.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.680 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.681 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.681 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.681 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612172+0000", 2023-11-01T04:17:09.681 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.682 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.682 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.682 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612172+0000", 2023-11-01T04:17:09.682 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.682 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.683 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.683 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612173+0000", 2023-11-01T04:17:09.683 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4300000000000002e-07 2023-11-01T04:17:09.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.683 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.684 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.684 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612179+0000", 2023-11-01T04:17:09.684 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.9699999999999996e-06 2023-11-01T04:17:09.684 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.684 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.684 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.685 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.612190+0000", 2023-11-01T04:17:09.685 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1790000000000001e-05 2023-11-01T04:17:09.685 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.685 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.685 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.685 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.712330+0000", 2023-11-01T04:17:09.686 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10013968099999999 2023-11-01T04:17:09.686 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.686 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.686 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.686 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.716489+0000", 2023-11-01T04:17:09.687 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0041593480000000002 2023-11-01T04:17:09.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.687 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.687 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.687 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.716832+0000", 2023-11-01T04:17:09.688 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00034265100000000002 2023-11-01T04:17:09.688 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.688 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.688 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:09.688 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.718665+0000", 2023-11-01T04:17:09.688 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001832752 2023-11-01T04:17:09.689 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.689 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.689 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:09.689 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.724658+0000", 2023-11-01T04:17:09.689 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0059932750000000002 2023-11-01T04:17:09.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.690 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.690 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:09.690 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.421975+0000", 2023-11-01T04:17:09.690 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.69731722500000004 2023-11-01T04:17:09.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.691 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.691 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.691 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.422349+0000", 2023-11-01T04:17:09.691 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000373884 2023-11-01T04:17:09.691 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.692 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.692 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.692 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.692 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.692 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:724 3.ds1 3:b0d0a55f:::smithi19039307-8:head [append 0~0] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:09.692 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:03.712366+0000", 2023-11-01T04:17:09.693 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 5.7436522329999997, 2023-11-01T04:17:09.693 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.81005693400000001, 2023-11-01T04:17:09.693 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.693 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:09.693 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.694 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.694 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.694 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 724 2023-11-01T04:17:09.694 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.694 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.694 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.695 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.695 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.712366+0000", 2023-11-01T04:17:09.695 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.695 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.695 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.695 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.696 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.712366+0000", 2023-11-01T04:17:09.696 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.696 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.696 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.696 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.696 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.712368+0000", 2023-11-01T04:17:09.697 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5799999999999999e-06 2023-11-01T04:17:09.697 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.697 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.697 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.697 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.712376+0000", 2023-11-01T04:17:09.697 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1669999999999999e-06 2023-11-01T04:17:09.698 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.698 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.698 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.698 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.712394+0000", 2023-11-01T04:17:09.698 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8576e-05 2023-11-01T04:17:09.698 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.699 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.699 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.699 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.712405+0000", 2023-11-01T04:17:09.699 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1039000000000001e-05 2023-11-01T04:17:09.699 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.699 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.700 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.700 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.721283+0000", 2023-11-01T04:17:09.700 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0088778039999999996 2023-11-01T04:17:09.700 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.700 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.721604+0000", 2023-11-01T04:17:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00032106600000000001 2023-11-01T04:17:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.701 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.723116+0000", 2023-11-01T04:17:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001512132 2023-11-01T04:17:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.702 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.725883+0000", 2023-11-01T04:17:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0027660720000000001 2023-11-01T04:17:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:09.703 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.522124+0000", 2023-11-01T04:17:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.79624189899999998 2023-11-01T04:17:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.522423+0000", 2023-11-01T04:17:09.704 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00029859899999999998 2023-11-01T04:17:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.705 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:778 3.as1 3:54622f20:::smithi19039307-19:head [read 0~1802240,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2023-11-01T04:17:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:08.964668+0000", 2023-11-01T04:17:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.49135005700000001, 2023-11-01T04:17:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.311844334, 2023-11-01T04:17:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2023-11-01T04:17:09.706 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 778 2023-11-01T04:17:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:09.707 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964668+0000", 2023-11-01T04:17:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.708 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964668+0000", 2023-11-01T04:17:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.709 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964669+0000", 2023-11-01T04:17:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.093e-06 2023-11-01T04:17:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.710 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964676+0000", 2023-11-01T04:17:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7059999999999998e-06 2023-11-01T04:17:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:09.711 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964694+0000", 2023-11-01T04:17:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8450000000000001e-05 2023-11-01T04:17:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:09.712 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964709+0000", 2023-11-01T04:17:09.713 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4518999999999999e-05 2023-11-01T04:17:09.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.713 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.713 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.713 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964910+0000", 2023-11-01T04:17:09.713 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00020095700000000001 2023-11-01T04:17:09.714 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.714 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.714 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:09.714 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:08.964921+0000", 2023-11-01T04:17:09.714 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1177e-05 2023-11-01T04:17:09.714 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.715 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.715 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:09.715 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.169822+0000", 2023-11-01T04:17:09.715 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.204900891 2023-11-01T04:17:09.715 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.716 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.716 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:09.716 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.169903+0000", 2023-11-01T04:17:09.716 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1149999999999994e-05 2023-11-01T04:17:09.716 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:09.716 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:09.717 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:09.717 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:09.276513+0000", 2023-11-01T04:17:09.717 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.106609391 2023-11-01T04:17:09.717 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.717 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.717 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.718 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:09.718 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:09.718 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:17:09.720 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-01T04:17:09.720 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:09.718+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:09.820 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:17:09.821+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:17:09.894 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:17:09.895 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:17:09.895 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2023-11-01T04:17:09.895 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:17:09.908 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-01T04:17:09.919 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:09.964 INFO:teuthology.orchestra.run.smithi153.stderr:nodeep-scrub is set 2023-11-01T04:17:09.973 INFO:tasks.rados.rados.0.smithi190.stdout:319: done (6 left) 2023-11-01T04:17:09.973 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 16 current snap is 29 2023-11-01T04:17:09.973 INFO:tasks.rados.rados.0.smithi190.stdout:320: seq_num 118 ranges {6307840=442368,6750208=753664,7503872=655360,8159232=147456} 2023-11-01T04:17:10.022 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:10.018+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:10.032 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:17:10.032 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2023-11-01T04:17:10.033 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2023-11-01T04:17:10.033 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2023-11-01T04:17:10.033 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:17:10.046 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-01T04:17:10.059 INFO:tasks.rados.rados.0.smithi190.stdout:320: writing smithi19039307-16 from 6307840 to 6750208 tid 1 2023-11-01T04:17:10.064 INFO:tasks.rados.rados.0.smithi190.stdout:320: writing smithi19039307-16 from 6750208 to 7503872 tid 2 2023-11-01T04:17:10.069 INFO:tasks.rados.rados.0.smithi190.stdout:320: writing smithi19039307-16 from 7503872 to 8159232 tid 3 2023-11-01T04:17:10.070 INFO:tasks.rados.rados.0.smithi190.stdout:320: writing smithi19039307-16 from 8159232 to 8306688 tid 4 2023-11-01T04:17:10.070 INFO:tasks.rados.rados.0.smithi190.stdout:321: rollback oid 4 current snap is 29 2023-11-01T04:17:10.070 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 4 to 23 2023-11-01T04:17:10.071 INFO:tasks.rados.rados.0.smithi190.stdout:314: expect (ObjNum 0 snap 0 seq_num 0) 2023-11-01T04:17:10.072 INFO:tasks.rados.rados.0.smithi190.stdout:317: expect (ObjNum 103 snap 21 seq_num 103) 2023-11-01T04:17:10.122 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:17:10.119+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:17:10.169 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2023-11-01T04:17:10.169 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2023-11-01T04:17:10.169 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2023-11-01T04:17:10.169 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2023-11-01T04:17:10.170 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.170 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.3s1 36/31 ECSubWrite(tid=189, reqid=client.4559.0:151, at_version=36'6, trim_to=0'0, roll_forward_to=35'4))", 2023-11-01T04:17:10.170 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.417087+0000", 2023-11-01T04:17:10.170 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 54.750478393999998, 2023-11-01T04:17:10.170 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103983992, 2023-11-01T04:17:10.171 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.171 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.171 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.171 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.171 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.171 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417087+0000", 2023-11-01T04:17:10.172 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.172 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.172 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.172 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417087+0000", 2023-11-01T04:17:10.173 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.173 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.173 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.173 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417087+0000", 2023-11-01T04:17:10.173 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2499999999999995e-07 2023-11-01T04:17:10.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.174 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.174 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.174 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417161+0000", 2023-11-01T04:17:10.174 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3508000000000002e-05 2023-11-01T04:17:10.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.175 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.175 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.175 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417171+0000", 2023-11-01T04:17:10.175 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7010000000000008e-06 2023-11-01T04:17:10.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.176 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.176 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.176 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417180+0000", 2023-11-01T04:17:10.176 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2140000000000002e-06 2023-11-01T04:17:10.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.177 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.177 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.177 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.517702+0000", 2023-11-01T04:17:10.177 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100522186 2023-11-01T04:17:10.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.178 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.178 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:10.178 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.517709+0000", 2023-11-01T04:17:10.178 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0299999999999996e-06 2023-11-01T04:17:10.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.178 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.179 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:10.179 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.521045+0000", 2023-11-01T04:17:10.179 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0033360640000000001 2023-11-01T04:17:10.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.179 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.179 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.180 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.521071+0000", 2023-11-01T04:17:10.180 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5763999999999999e-05 2023-11-01T04:17:10.180 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.180 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.180 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.181 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.181 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.3s1 36/31 ECSubWrite(tid=190, reqid=client.4559.0:152, at_version=36'7, trim_to=0'0, roll_forward_to=35'4))", 2023-11-01T04:17:10.181 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:15.417198+0000", 2023-11-01T04:17:10.181 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 54.750367013999998, 2023-11-01T04:17:10.181 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103890102, 2023-11-01T04:17:10.182 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.182 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.182 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.182 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.182 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.182 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417198+0000", 2023-11-01T04:17:10.183 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.183 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.183 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.183 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417198+0000", 2023-11-01T04:17:10.183 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.184 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.184 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.184 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417199+0000", 2023-11-01T04:17:10.184 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1300000000000001e-07 2023-11-01T04:17:10.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.185 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.185 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.185 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417214+0000", 2023-11-01T04:17:10.185 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5702999999999999e-05 2023-11-01T04:17:10.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.186 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.186 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.186 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417226+0000", 2023-11-01T04:17:10.186 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1602e-05 2023-11-01T04:17:10.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.186 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.187 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.187 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.417238+0000", 2023-11-01T04:17:10.187 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2133000000000001e-05 2023-11-01T04:17:10.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.187 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.188 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.188 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.518027+0000", 2023-11-01T04:17:10.188 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100788603 2023-11-01T04:17:10.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.188 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.189 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:10.189 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.518038+0000", 2023-11-01T04:17:10.189 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0963e-05 2023-11-01T04:17:10.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.189 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:10.190 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.521074+0000", 2023-11-01T04:17:10.190 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0030366820000000002 2023-11-01T04:17:10.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.190 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.190 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.191 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:15.521088+0000", 2023-11-01T04:17:10.191 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4003e-05 2023-11-01T04:17:10.191 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.191 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.191 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.192 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s4 37/31 ECSubWrite(tid=207, reqid=client.4559.0:207, at_version=37'21, trim_to=0'0, roll_forward_to=36'16))", 2023-11-01T04:17:10.192 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:17.382276+0000", 2023-11-01T04:17:10.192 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 52.785288891, 2023-11-01T04:17:10.192 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103388864, 2023-11-01T04:17:10.193 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.193 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.193 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.193 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.194 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.382276+0000", 2023-11-01T04:17:10.194 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.194 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.194 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.195 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.382276+0000", 2023-11-01T04:17:10.195 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.195 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.195 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.195 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.382278+0000", 2023-11-01T04:17:10.196 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7630000000000001e-06 2023-11-01T04:17:10.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.196 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.196 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.196 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.382534+0000", 2023-11-01T04:17:10.197 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00025576899999999998 2023-11-01T04:17:10.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.197 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.197 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.197 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.382567+0000", 2023-11-01T04:17:10.198 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2719000000000003e-05 2023-11-01T04:17:10.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.198 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.198 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.198 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.482664+0000", 2023-11-01T04:17:10.199 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100097424 2023-11-01T04:17:10.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.199 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.199 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.199 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.482806+0000", 2023-11-01T04:17:10.200 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014185599999999999 2023-11-01T04:17:10.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.200 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.200 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:10.200 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.482822+0000", 2023-11-01T04:17:10.201 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5743000000000002e-05 2023-11-01T04:17:10.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.201 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.201 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:10.201 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.485590+0000", 2023-11-01T04:17:10.202 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0027687990000000002 2023-11-01T04:17:10.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.202 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.202 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.202 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:17.485665+0000", 2023-11-01T04:17:10.203 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4790999999999996e-05 2023-11-01T04:17:10.203 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.203 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.203 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.204 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.204 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 41/31 ECSubWrite(tid=55, reqid=client.4559.0:266, at_version=41'21, trim_to=0'0, roll_forward_to=37'19))", 2023-11-01T04:17:10.204 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.119591+0000", 2023-11-01T04:17:10.204 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 49.047973906999999, 2023-11-01T04:17:10.204 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.106535935, 2023-11-01T04:17:10.204 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.205 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.205 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.205 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.205 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.205 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.119591+0000", 2023-11-01T04:17:10.205 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.206 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.206 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.206 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.119591+0000", 2023-11-01T04:17:10.206 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.207 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.207 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.207 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.119593+0000", 2023-11-01T04:17:10.207 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9230000000000001e-06 2023-11-01T04:17:10.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.208 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.208 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.119864+0000", 2023-11-01T04:17:10.208 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000270623 2023-11-01T04:17:10.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.209 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.209 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.209 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.119910+0000", 2023-11-01T04:17:10.209 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6144999999999999e-05 2023-11-01T04:17:10.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.209 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.210 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.210 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.119934+0000", 2023-11-01T04:17:10.210 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3643e-05 2023-11-01T04:17:10.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.210 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.211 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.211 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.120148+0000", 2023-11-01T04:17:10.211 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000214224 2023-11-01T04:17:10.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.211 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.211 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:10.212 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.120173+0000", 2023-11-01T04:17:10.212 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4790000000000002e-05 2023-11-01T04:17:10.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.212 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.212 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:10.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.226085+0000", 2023-11-01T04:17:10.213 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10591191699999999 2023-11-01T04:17:10.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.213 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.213 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.226127+0000", 2023-11-01T04:17:10.214 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2670000000000003e-05 2023-11-01T04:17:10.214 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.214 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.214 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.215 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.215 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 41/31 ECSubWrite(tid=56, reqid=client.4559.0:267, at_version=41'22, trim_to=0'0, roll_forward_to=37'19))", 2023-11-01T04:17:10.215 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.121320+0000", 2023-11-01T04:17:10.215 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 49.046245282000001, 2023-11-01T04:17:10.215 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104845891, 2023-11-01T04:17:10.215 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.216 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.216 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.216 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.216 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.216 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.121320+0000", 2023-11-01T04:17:10.216 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.217 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.217 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.217 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.121320+0000", 2023-11-01T04:17:10.217 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.218 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.218 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.218 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.121322+0000", 2023-11-01T04:17:10.218 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0360000000000001e-06 2023-11-01T04:17:10.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.219 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.219 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.219 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.121511+0000", 2023-11-01T04:17:10.219 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000188612 2023-11-01T04:17:10.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.219 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.220 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.220 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.121545+0000", 2023-11-01T04:17:10.220 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4269e-05 2023-11-01T04:17:10.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.220 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.221 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.221 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.121571+0000", 2023-11-01T04:17:10.221 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6043000000000001e-05 2023-11-01T04:17:10.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.221 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.222 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122494+0000", 2023-11-01T04:17:10.222 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00092303199999999995 2023-11-01T04:17:10.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.222 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.222 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:10.223 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122518+0000", 2023-11-01T04:17:10.223 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4036000000000002e-05 2023-11-01T04:17:10.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.223 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.223 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:10.223 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.226144+0000", 2023-11-01T04:17:10.224 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.103625515 2023-11-01T04:17:10.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.224 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.224 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.224 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.226166+0000", 2023-11-01T04:17:10.224 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2348e-05 2023-11-01T04:17:10.225 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.225 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.225 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.225 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.226 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 41/31 ECSubWrite(tid=57, reqid=client.4559.0:268, at_version=41'23, trim_to=0'0, roll_forward_to=37'19))", 2023-11-01T04:17:10.226 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.122597+0000", 2023-11-01T04:17:10.226 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 49.044968329, 2023-11-01T04:17:10.226 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.106843242, 2023-11-01T04:17:10.226 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.227 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.227 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.227 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.227 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.227 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122597+0000", 2023-11-01T04:17:10.228 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.228 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.228 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.228 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122597+0000", 2023-11-01T04:17:10.228 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.229 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.229 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.229 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122599+0000", 2023-11-01T04:17:10.229 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6729999999999999e-06 2023-11-01T04:17:10.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.230 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.230 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.230 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122640+0000", 2023-11-01T04:17:10.230 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1768999999999999e-05 2023-11-01T04:17:10.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.231 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.231 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.231 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122673+0000", 2023-11-01T04:17:10.231 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2202000000000001e-05 2023-11-01T04:17:10.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.232 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.232 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.232 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.122695+0000", 2023-11-01T04:17:10.232 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2657e-05 2023-11-01T04:17:10.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.232 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.233 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.233 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.224915+0000", 2023-11-01T04:17:10.233 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10221991599999999 2023-11-01T04:17:10.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.233 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.233 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:10.234 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.224950+0000", 2023-11-01T04:17:10.234 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4789000000000002e-05 2023-11-01T04:17:10.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.234 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.234 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:10.234 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.229406+0000", 2023-11-01T04:17:10.235 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0044560440000000002 2023-11-01T04:17:10.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.235 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.235 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.235 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.229440+0000", 2023-11-01T04:17:10.235 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4192000000000001e-05 2023-11-01T04:17:10.236 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.236 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.236 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.236 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.237 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 41/31 ECSubWrite(tid=58, reqid=osd.12.0:58, at_version=41'25, trim_to=0'0, roll_forward_to=37'19))", 2023-11-01T04:17:10.237 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:21.123645+0000", 2023-11-01T04:17:10.237 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 49.043919956000003, 2023-11-01T04:17:10.237 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.105819151, 2023-11-01T04:17:10.237 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.237 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.238 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.238 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.238 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.238 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123645+0000", 2023-11-01T04:17:10.238 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.239 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.239 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.239 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123645+0000", 2023-11-01T04:17:10.239 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.240 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.240 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.240 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123647+0000", 2023-11-01T04:17:10.240 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.716e-06 2023-11-01T04:17:10.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.240 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.241 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.241 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123698+0000", 2023-11-01T04:17:10.241 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1474999999999998e-05 2023-11-01T04:17:10.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.241 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.241 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.242 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.123730+0000", 2023-11-01T04:17:10.242 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1600000000000002e-05 2023-11-01T04:17:10.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.242 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.242 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.242 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.223820+0000", 2023-11-01T04:17:10.243 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100089995 2023-11-01T04:17:10.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.243 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.243 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.243 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.225389+0000", 2023-11-01T04:17:10.243 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0015688150000000001 2023-11-01T04:17:10.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.244 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.244 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:10.244 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.225407+0000", 2023-11-01T04:17:10.244 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7980000000000001e-05 2023-11-01T04:17:10.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.245 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.245 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:10.245 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.229444+0000", 2023-11-01T04:17:10.245 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0040372419999999999 2023-11-01T04:17:10.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.245 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.246 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.246 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:21.229464+0000", 2023-11-01T04:17:10.246 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0327999999999999e-05 2023-11-01T04:17:10.246 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.246 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.246 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.247 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.247 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.2s4 46/31 ECSubWrite(tid=236, reqid=client.4559.0:330, at_version=46'36, trim_to=0'0, roll_forward_to=45'33))", 2023-11-01T04:17:10.247 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:26.282877+0000", 2023-11-01T04:17:10.247 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 43.884687835999998, 2023-11-01T04:17:10.247 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20107656700000001, 2023-11-01T04:17:10.248 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.248 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.248 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.248 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.248 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.248 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282877+0000", 2023-11-01T04:17:10.249 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.249 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.249 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.249 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.249 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282877+0000", 2023-11-01T04:17:10.250 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.250 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.250 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.250 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.282880+0000", 2023-11-01T04:17:10.250 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.357e-06 2023-11-01T04:17:10.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.251 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.251 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.283033+0000", 2023-11-01T04:17:10.251 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015317900000000001 2023-11-01T04:17:10.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.252 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.252 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.252 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.283074+0000", 2023-11-01T04:17:10.252 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0942999999999997e-05 2023-11-01T04:17:10.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.253 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.253 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383201+0000", 2023-11-01T04:17:10.253 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100126696 2023-11-01T04:17:10.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.254 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.254 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383268+0000", 2023-11-01T04:17:10.254 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7747e-05 2023-11-01T04:17:10.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.254 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:10.255 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.383280+0000", 2023-11-01T04:17:10.255 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1242e-05 2023-11-01T04:17:10.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.255 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.255 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:10.256 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.483899+0000", 2023-11-01T04:17:10.256 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10061980399999999 2023-11-01T04:17:10.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.256 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.256 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.256 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:26.483954+0000", 2023-11-01T04:17:10.257 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4598999999999997e-05 2023-11-01T04:17:10.257 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.257 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.257 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.257 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.258 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.3s1 58/31 ECSubWrite(tid=279, reqid=osd.13.0:279, at_version=58'37, trim_to=0'0, roll_forward_to=57'33))", 2023-11-01T04:17:10.258 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:38.143156+0000", 2023-11-01T04:17:10.258 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 32.024409499999997, 2023-11-01T04:17:10.258 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104186716, 2023-11-01T04:17:10.258 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.258 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.259 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.259 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.259 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.259 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.143156+0000", 2023-11-01T04:17:10.259 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.259 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.260 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.260 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.260 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.143156+0000", 2023-11-01T04:17:10.260 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.260 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.261 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.261 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.143157+0000", 2023-11-01T04:17:10.261 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.573e-06 2023-11-01T04:17:10.261 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.261 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.261 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.262 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.143208+0000", 2023-11-01T04:17:10.262 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0283e-05 2023-11-01T04:17:10.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.262 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.262 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.263 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.143237+0000", 2023-11-01T04:17:10.263 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9632999999999999e-05 2023-11-01T04:17:10.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.263 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.263 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.263 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243350+0000", 2023-11-01T04:17:10.264 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10011239399999999 2023-11-01T04:17:10.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.264 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.264 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.264 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243546+0000", 2023-11-01T04:17:10.264 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000196546 2023-11-01T04:17:10.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.265 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.265 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:10.265 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243574+0000", 2023-11-01T04:17:10.265 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7372000000000002e-05 2023-11-01T04:17:10.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.266 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.266 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:10.266 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.247245+0000", 2023-11-01T04:17:10.266 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0036718900000000001 2023-11-01T04:17:10.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.266 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.267 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.267 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.247342+0000", 2023-11-01T04:17:10.267 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7025e-05 2023-11-01T04:17:10.267 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.267 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.267 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.268 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.268 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 58/31 ECSubWrite(tid=109, reqid=osd.12.0:109, at_version=58'62, trim_to=0'0, roll_forward_to=57'58))", 2023-11-01T04:17:10.268 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:38.243889+0000", 2023-11-01T04:17:10.268 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 31.923675905, 2023-11-01T04:17:10.268 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.105244037, 2023-11-01T04:17:10.269 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.269 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.269 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.269 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.269 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243889+0000", 2023-11-01T04:17:10.270 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.270 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.270 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.270 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243889+0000", 2023-11-01T04:17:10.270 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.271 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.271 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.271 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243890+0000", 2023-11-01T04:17:10.271 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0100000000000001e-06 2023-11-01T04:17:10.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.272 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.272 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243930+0000", 2023-11-01T04:17:10.272 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0000000000000003e-05 2023-11-01T04:17:10.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.273 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.273 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.273 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243945+0000", 2023-11-01T04:17:10.273 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4473e-05 2023-11-01T04:17:10.273 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.273 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.274 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.274 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.243959+0000", 2023-11-01T04:17:10.274 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3763e-05 2023-11-01T04:17:10.274 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.274 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.274 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.275 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.245005+0000", 2023-11-01T04:17:10.275 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00104667 2023-11-01T04:17:10.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.275 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.275 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:10.276 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.245035+0000", 2023-11-01T04:17:10.276 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9411999999999999e-05 2023-11-01T04:17:10.276 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.276 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.276 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:10.277 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.349054+0000", 2023-11-01T04:17:10.277 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10401964499999999 2023-11-01T04:17:10.277 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.277 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.277 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.278 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.349133+0000", 2023-11-01T04:17:10.278 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9064000000000001e-05 2023-11-01T04:17:10.278 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.278 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.278 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.279 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 58/31 ECSubWrite(tid=111, reqid=osd.12.0:111, at_version=58'64, trim_to=0'0, roll_forward_to=58'62))", 2023-11-01T04:17:10.279 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:38.360098+0000", 2023-11-01T04:17:10.279 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 31.807467241000001, 2023-11-01T04:17:10.279 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20292976700000001, 2023-11-01T04:17:10.280 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.280 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.280 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.280 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.280 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.281 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.360098+0000", 2023-11-01T04:17:10.281 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.281 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.281 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.281 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.360098+0000", 2023-11-01T04:17:10.282 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.282 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.282 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.282 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.360100+0000", 2023-11-01T04:17:10.283 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7799999999999999e-06 2023-11-01T04:17:10.283 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.283 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.283 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.283 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.360153+0000", 2023-11-01T04:17:10.283 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2722000000000001e-05 2023-11-01T04:17:10.284 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.284 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.284 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.284 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.360195+0000", 2023-11-01T04:17:10.284 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2905999999999999e-05 2023-11-01T04:17:10.284 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.285 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.285 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.285 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.460529+0000", 2023-11-01T04:17:10.285 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10033367 2023-11-01T04:17:10.285 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.286 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.286 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.286 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.560802+0000", 2023-11-01T04:17:10.286 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100272668 2023-11-01T04:17:10.286 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.287 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.287 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:10.287 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.560837+0000", 2023-11-01T04:17:10.287 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5589000000000001e-05 2023-11-01T04:17:10.287 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.288 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.288 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:10.288 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.562956+0000", 2023-11-01T04:17:10.288 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0021184530000000002 2023-11-01T04:17:10.288 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.289 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.289 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.289 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:38.563028+0000", 2023-11-01T04:17:10.289 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1978999999999996e-05 2023-11-01T04:17:10.289 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.289 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.290 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.290 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.290 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.290 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 68/31 ECSubWriteReply(tid=11, last_complete=68'10, committed=1, applied=1))", 2023-11-01T04:17:10.290 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:47.995605+0000", 2023-11-01T04:17:10.291 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 22.171959843, 2023-11-01T04:17:10.291 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.199563451, 2023-11-01T04:17:10.291 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.291 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.291 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.291 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.292 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.292 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.995605+0000", 2023-11-01T04:17:10.292 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.292 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.292 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.292 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.293 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.995605+0000", 2023-11-01T04:17:10.293 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.293 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.293 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.293 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.995606+0000", 2023-11-01T04:17:10.294 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8999999999999996e-07 2023-11-01T04:17:10.294 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.294 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.294 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.294 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.995609+0000", 2023-11-01T04:17:10.295 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2729999999999998e-06 2023-11-01T04:17:10.295 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.295 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.295 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.295 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.995619+0000", 2023-11-01T04:17:10.295 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4630000000000007e-06 2023-11-01T04:17:10.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.296 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.296 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.296 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:47.995625+0000", 2023-11-01T04:17:10.296 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2879999999999998e-06 2023-11-01T04:17:10.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.297 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.297 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.297 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.195112+0000", 2023-11-01T04:17:10.297 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19948711799999999 2023-11-01T04:17:10.297 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.298 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.298 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.298 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.195169+0000", 2023-11-01T04:17:10.298 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6718999999999997e-05 2023-11-01T04:17:10.298 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.298 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.299 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.299 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.299 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.299 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 68/31 ECSubWriteReply(tid=11, last_complete=68'10, committed=1, applied=1))", 2023-11-01T04:17:10.299 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:48.009864+0000", 2023-11-01T04:17:10.299 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 22.157701005, 2023-11-01T04:17:10.300 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.18553544199999999, 2023-11-01T04:17:10.300 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.300 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.300 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.300 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.300 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.301 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.009864+0000", 2023-11-01T04:17:10.301 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.301 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.301 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.301 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.301 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.009864+0000", 2023-11-01T04:17:10.302 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.302 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.302 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.302 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.302 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.009865+0000", 2023-11-01T04:17:10.302 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.8100000000000001e-07 2023-11-01T04:17:10.303 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.303 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.303 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.303 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.009870+0000", 2023-11-01T04:17:10.303 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.442e-06 2023-11-01T04:17:10.303 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.304 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.304 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.304 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.009885+0000", 2023-11-01T04:17:10.304 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5192e-05 2023-11-01T04:17:10.304 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.305 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.305 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.305 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.110017+0000", 2023-11-01T04:17:10.305 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10013217100000001 2023-11-01T04:17:10.305 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.305 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.306 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.306 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.195366+0000", 2023-11-01T04:17:10.306 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.085349353000000003 2023-11-01T04:17:10.306 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.306 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.306 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.307 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:48.195400+0000", 2023-11-01T04:17:10.307 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.3303000000000003e-05 2023-11-01T04:17:10.307 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.307 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.307 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.307 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.308 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.308 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 69/68 ECSubWrite(tid=2, reqid=client.4559.0:581, at_version=69'77, trim_to=0'0, roll_forward_to=0'0))", 2023-11-01T04:17:10.308 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:16:50.229507+0000", 2023-11-01T04:17:10.308 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 19.938058057999999, 2023-11-01T04:17:10.308 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10351554, 2023-11-01T04:17:10.308 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.309 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.309 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.309 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.309 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.309 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.229507+0000", 2023-11-01T04:17:10.309 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.310 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.310 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.310 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.229507+0000", 2023-11-01T04:17:10.310 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.311 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.311 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.311 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.229508+0000", 2023-11-01T04:17:10.311 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.7500000000000001e-07 2023-11-01T04:17:10.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.311 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.312 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.312 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.229529+0000", 2023-11-01T04:17:10.312 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1217999999999999e-05 2023-11-01T04:17:10.312 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.312 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.312 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.313 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.229537+0000", 2023-11-01T04:17:10.313 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.2060000000000007e-06 2023-11-01T04:17:10.313 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.313 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.313 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.314 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.329628+0000", 2023-11-01T04:17:10.314 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100091236 2023-11-01T04:17:10.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.314 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.314 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.314 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.329811+0000", 2023-11-01T04:17:10.315 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000182738 2023-11-01T04:17:10.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.315 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.315 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:10.315 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.329838+0000", 2023-11-01T04:17:10.315 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6764999999999999e-05 2023-11-01T04:17:10.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.316 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.316 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:10.316 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.332958+0000", 2023-11-01T04:17:10.316 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00311984 2023-11-01T04:17:10.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.317 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.317 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.317 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:16:50.333023+0000", 2023-11-01T04:17:10.317 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5161999999999997e-05 2023-11-01T04:17:10.317 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.317 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.318 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.318 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.318 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.318 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWrite(3.ds4 82/68 ECSubWrite(tid=35, reqid=client.4559.0:723, at_version=82'97, trim_to=0'0, roll_forward_to=79'94))", 2023-11-01T04:17:10.318 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:03.718778+0000", 2023-11-01T04:17:10.319 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.4487872660000001, 2023-11-01T04:17:10.319 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.104317588, 2023-11-01T04:17:10.319 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.319 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.319 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.320 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.320 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.320 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.718778+0000", 2023-11-01T04:17:10.320 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.320 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.321 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.321 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.718778+0000", 2023-11-01T04:17:10.321 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.321 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.322 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.322 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.718780+0000", 2023-11-01T04:17:10.322 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8369999999999999e-06 2023-11-01T04:17:10.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.322 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.322 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.323 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.718856+0000", 2023-11-01T04:17:10.323 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6580000000000002e-05 2023-11-01T04:17:10.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.323 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.323 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.323 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.718891+0000", 2023-11-01T04:17:10.324 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4112000000000001e-05 2023-11-01T04:17:10.324 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.324 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.324 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.324 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.819030+0000", 2023-11-01T04:17:10.324 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.100139491 2023-11-01T04:17:10.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.325 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.325 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.819202+0000", 2023-11-01T04:17:10.325 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00017185400000000001 2023-11-01T04:17:10.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.326 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.326 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:10.326 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.819247+0000", 2023-11-01T04:17:10.326 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4792000000000001e-05 2023-11-01T04:17:10.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.326 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.327 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:10.327 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.823016+0000", 2023-11-01T04:17:10.327 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.003769002 2023-11-01T04:17:10.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.327 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.328 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.328 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.823096+0000", 2023-11-01T04:17:10.328 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9919999999999994e-05 2023-11-01T04:17:10.328 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.328 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.328 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.329 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.329 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:744 3.5s0 3:a1a35b6e:::smithi19039307-21:head [append 0~0] snapc 30=[30,2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2023-11-01T04:17:10.329 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:03.912324+0000", 2023-11-01T04:17:10.329 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.2552412410000002, 2023-11-01T04:17:10.329 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.50967580499999998, 2023-11-01T04:17:10.330 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.330 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:10.330 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:10.330 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:10.330 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:10.330 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 744 2023-11-01T04:17:10.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.331 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.331 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.331 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912324+0000", 2023-11-01T04:17:10.332 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.332 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.332 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.332 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912324+0000", 2023-11-01T04:17:10.332 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.333 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.333 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.333 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912327+0000", 2023-11-01T04:17:10.333 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8720000000000002e-06 2023-11-01T04:17:10.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.334 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.334 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912339+0000", 2023-11-01T04:17:10.334 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2024e-05 2023-11-01T04:17:10.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.335 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.335 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912368+0000", 2023-11-01T04:17:10.335 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8637e-05 2023-11-01T04:17:10.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.335 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.335 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.336 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912383+0000", 2023-11-01T04:17:10.336 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5034e-05 2023-11-01T04:17:10.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.336 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.336 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.336 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912510+0000", 2023-11-01T04:17:10.337 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012780300000000001 2023-11-01T04:17:10.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.337 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2023-11-01T04:17:10.337 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912531+0000", 2023-11-01T04:17:10.337 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1047999999999999e-05 2023-11-01T04:17:10.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.338 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.338 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.916058+0000", 2023-11-01T04:17:10.338 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0035268529999999999 2023-11-01T04:17:10.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.339 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:10.339 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.916245+0000", 2023-11-01T04:17:10.339 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00018656799999999999 2023-11-01T04:17:10.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.340 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:10.340 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.917640+0000", 2023-11-01T04:17:10.340 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001395479 2023-11-01T04:17:10.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.341 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:10.341 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.922753+0000", 2023-11-01T04:17:10.341 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0051131400000000004 2023-11-01T04:17:10.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.341 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:10.342 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.421842+0000", 2023-11-01T04:17:10.342 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.49908834899999999 2023-11-01T04:17:10.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.342 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.342 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.343 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.422000+0000", 2023-11-01T04:17:10.343 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015799799999999999 2023-11-01T04:17:10.343 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.343 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.343 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.344 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.344 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:745 3.5s0 3:a1a35b6e:::smithi19039307-21:head [rollback 2f] snapc 30=[30,2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2023-11-01T04:17:10.344 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:03.912416+0000", 2023-11-01T04:17:10.344 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.2551495570000002, 2023-11-01T04:17:10.344 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.61003988200000003, 2023-11-01T04:17:10.344 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.345 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:10.345 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:10.345 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:10.345 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:10.345 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 745 2023-11-01T04:17:10.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.346 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.346 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.346 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.346 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912416+0000", 2023-11-01T04:17:10.346 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.347 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.347 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912416+0000", 2023-11-01T04:17:10.347 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.348 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.348 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.348 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912416+0000", 2023-11-01T04:17:10.348 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.8100000000000002e-07 2023-11-01T04:17:10.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.348 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.349 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.349 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912420+0000", 2023-11-01T04:17:10.349 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8539999999999999e-06 2023-11-01T04:17:10.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.349 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.350 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912436+0000", 2023-11-01T04:17:10.350 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5784999999999999e-05 2023-11-01T04:17:10.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.350 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.351 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912456+0000", 2023-11-01T04:17:10.351 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9979e-05 2023-11-01T04:17:10.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.351 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.351 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912677+0000", 2023-11-01T04:17:10.352 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000220459 2023-11-01T04:17:10.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.352 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.352 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:10.352 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912694+0000", 2023-11-01T04:17:10.353 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7648000000000001e-05 2023-11-01T04:17:10.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.353 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.353 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.353 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.918929+0000", 2023-11-01T04:17:10.353 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0062346049999999998 2023-11-01T04:17:10.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.354 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.354 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:10.354 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.919186+0000", 2023-11-01T04:17:10.354 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000257626 2023-11-01T04:17:10.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.355 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.355 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:10.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.920361+0000", 2023-11-01T04:17:10.355 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001174554 2023-11-01T04:17:10.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.356 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.356 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:10.356 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.922813+0000", 2023-11-01T04:17:10.356 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0024521259999999998 2023-11-01T04:17:10.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.356 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.357 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:10.357 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.522208+0000", 2023-11-01T04:17:10.357 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.59939537700000001 2023-11-01T04:17:10.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.357 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.357 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.358 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.522456+0000", 2023-11-01T04:17:10.358 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00024708800000000002 2023-11-01T04:17:10.358 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.358 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.358 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.359 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.359 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.4559.0:746 3.5s0 3:a1a35b6e:::smithi19039307-21:head [append 0~0] snapc 30=[30,2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2023-11-01T04:17:10.359 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:03.912472+0000", 2023-11-01T04:17:10.359 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.2550933239999997, 2023-11-01T04:17:10.359 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.61048543499999997, 2023-11-01T04:17:10.359 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.360 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:10.360 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2023-11-01T04:17:10.360 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.4559", 2023-11-01T04:17:10.360 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:10.360 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 746 2023-11-01T04:17:10.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.361 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.361 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.361 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912472+0000", 2023-11-01T04:17:10.362 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.362 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.362 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912472+0000", 2023-11-01T04:17:10.363 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.363 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.363 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912473+0000", 2023-11-01T04:17:10.363 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3399999999999998e-07 2023-11-01T04:17:10.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.364 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.364 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912476+0000", 2023-11-01T04:17:10.364 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.3629999999999998e-06 2023-11-01T04:17:10.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.365 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.365 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.365 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912504+0000", 2023-11-01T04:17:10.365 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7804999999999999e-05 2023-11-01T04:17:10.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.366 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.366 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912512+0000", 2023-11-01T04:17:10.366 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.2099999999999993e-06 2023-11-01T04:17:10.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.367 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.367 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912736+0000", 2023-11-01T04:17:10.367 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00022369499999999999 2023-11-01T04:17:10.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.368 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:10.368 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.912741+0000", 2023-11-01T04:17:10.368 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0130000000000001e-06 2023-11-01T04:17:10.368 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.368 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.368 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.369 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.921213+0000", 2023-11-01T04:17:10.369 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0084724780000000003 2023-11-01T04:17:10.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.369 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.369 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2023-11-01T04:17:10.370 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.921387+0000", 2023-11-01T04:17:10.370 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000173707 2023-11-01T04:17:10.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.370 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.370 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_started", 2023-11-01T04:17:10.370 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.922200+0000", 2023-11-01T04:17:10.371 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00081335100000000003 2023-11-01T04:17:10.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.371 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.371 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_committed", 2023-11-01T04:17:10.371 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:03.924722+0000", 2023-11-01T04:17:10.372 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0025212910000000002 2023-11-01T04:17:10.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.372 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.372 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2023-11-01T04:17:10.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.522732+0000", 2023-11-01T04:17:10.373 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.59800996900000003 2023-11-01T04:17:10.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.373 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.522957+0000", 2023-11-01T04:17:10.373 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00022581899999999999 2023-11-01T04:17:10.374 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.374 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.374 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.374 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.374 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 83/31 ECSubWriteReply(tid=20, last_complete=83'20, committed=1, applied=1))", 2023-11-01T04:17:10.375 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:04.022855+0000", 2023-11-01T04:17:10.375 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.1447099740000004, 2023-11-01T04:17:10.375 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10343382600000001, 2023-11-01T04:17:10.375 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.375 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.375 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.376 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.376 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.376 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.022855+0000", 2023-11-01T04:17:10.376 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.376 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.377 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.377 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.022855+0000", 2023-11-01T04:17:10.377 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.377 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.378 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.022858+0000", 2023-11-01T04:17:10.378 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.294e-06 2023-11-01T04:17:10.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.378 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.378 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.022869+0000", 2023-11-01T04:17:10.379 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1883e-05 2023-11-01T04:17:10.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.379 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.379 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.022938+0000", 2023-11-01T04:17:10.380 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8479999999999995e-05 2023-11-01T04:17:10.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.380 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.380 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.380 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.022957+0000", 2023-11-01T04:17:10.381 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9137e-05 2023-11-01T04:17:10.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.381 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.381 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.381 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.126248+0000", 2023-11-01T04:17:10.381 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10329083999999999 2023-11-01T04:17:10.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.382 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.382 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.382 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.126289+0000", 2023-11-01T04:17:10.382 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1192000000000001e-05 2023-11-01T04:17:10.382 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.383 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.383 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.383 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.383 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 83/31 ECSubWriteReply(tid=22, last_complete=83'22, committed=1, applied=1))", 2023-11-01T04:17:10.384 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2023-11-01T04:17:04.023287+0000", 2023-11-01T04:17:10.384 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 6.1442782979999997, 2023-11-01T04:17:10.384 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10343377500000001, 2023-11-01T04:17:10.384 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2023-11-01T04:17:10.384 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2023-11-01T04:17:10.385 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2023-11-01T04:17:10.385 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.385 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2023-11-01T04:17:10.385 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.023287+0000", 2023-11-01T04:17:10.385 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.386 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.386 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2023-11-01T04:17:10.386 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.023287+0000", 2023-11-01T04:17:10.386 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2023-11-01T04:17:10.386 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.387 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.387 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2023-11-01T04:17:10.387 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.023288+0000", 2023-11-01T04:17:10.387 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2330000000000001e-06 2023-11-01T04:17:10.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.388 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.388 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2023-11-01T04:17:10.388 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.023295+0000", 2023-11-01T04:17:10.388 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9460000000000001e-06 2023-11-01T04:17:10.388 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.388 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.389 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2023-11-01T04:17:10.389 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.023314+0000", 2023-11-01T04:17:10.389 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.933e-05 2023-11-01T04:17:10.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.389 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.390 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2023-11-01T04:17:10.390 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.023325+0000", 2023-11-01T04:17:10.390 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0166000000000001e-05 2023-11-01T04:17:10.390 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.390 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.391 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2023-11-01T04:17:10.391 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.126665+0000", 2023-11-01T04:17:10.391 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.10333988199999999 2023-11-01T04:17:10.391 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2023-11-01T04:17:10.391 INFO:teuthology.orchestra.run.smithi153.stdout: { 2023-11-01T04:17:10.392 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2023-11-01T04:17:10.392 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2023-11-01T04:17:04.126721+0000", 2023-11-01T04:17:10.392 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6218000000000003e-05 2023-11-01T04:17:10.392 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.392 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.392 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.393 INFO:teuthology.orchestra.run.smithi153.stdout: } 2023-11-01T04:17:10.393 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2023-11-01T04:17:10.393 INFO:teuthology.orchestra.run.smithi153.stdout:} 2023-11-01T04:17:10.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-01T04:17:10.396 INFO:tasks.rados.rados.0.smithi190.stdout:318: expect (ObjNum 89 snap 20 seq_num 89) 2023-11-01T04:17:10.396 INFO:tasks.rados.rados.0.smithi190.stdout:313: expect (ObjNum 112 snap 25 seq_num 112) 2023-11-01T04:17:10.396 INFO:tasks.rados.rados.0.smithi190.stdout:315: expect (ObjNum 66 snap 6 seq_num 66) 2023-11-01T04:17:10.396 INFO:tasks.rados.rados.0.smithi190.stdout:316: finishing copy_from racing read to smithi19039307-23 2023-11-01T04:17:10.396 INFO:tasks.rados.rados.0.smithi190.stdout:316: finishing copy_from to smithi19039307-23 2023-11-01T04:17:10.397 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 23 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2023-11-01T04:17:10.397 INFO:tasks.rados.rados.0.smithi190.stdout:313: done (7 left) 2023-11-01T04:17:10.397 INFO:tasks.rados.rados.0.smithi190.stdout:314: done (6 left) 2023-11-01T04:17:10.397 INFO:tasks.rados.rados.0.smithi190.stdout:315: done (5 left) 2023-11-01T04:17:10.397 INFO:tasks.rados.rados.0.smithi190.stdout:316: done (4 left) 2023-11-01T04:17:10.398 INFO:tasks.rados.rados.0.smithi190.stdout:317: done (3 left) 2023-11-01T04:17:10.398 INFO:tasks.rados.rados.0.smithi190.stdout:318: done (2 left) 2023-11-01T04:17:10.398 INFO:tasks.rados.rados.0.smithi190.stdout:322: snap_create 2023-11-01T04:17:10.398 INFO:tasks.rados.rados.0.smithi190.stdout:321: finishing rollback tid 0 to smithi19039307-4 2023-11-01T04:17:10.398 INFO:tasks.rados.rados.0.smithi190.stdout:321: finishing rollback tid 1 to smithi19039307-4 2023-11-01T04:17:10.399 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:17:10.393+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:17:10.496 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:17:10.497+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:17:10.551 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-01T04:17:10.552 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:10.554 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:10.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-01T04:17:10.581 INFO:tasks.rados.rados.0.smithi190.stdout:321: finishing rollback tid 2 to smithi19039307-4 2023-11-01T04:17:10.582 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 4 v 71 (ObjNum 3 snap 0 seq_num 3) dirty exists 2023-11-01T04:17:10.596 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:10.594+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:10.675 INFO:tasks.rados.rados.0.smithi190.stdout:320: finishing write tid 1 to smithi19039307-16 2023-11-01T04:17:10.676 INFO:tasks.rados.rados.0.smithi190.stdout:320: finishing write tid 2 to smithi19039307-16 2023-11-01T04:17:10.677 INFO:tasks.rados.rados.0.smithi190.stdout:320: finishing write tid 3 to smithi19039307-16 2023-11-01T04:17:10.678 INFO:tasks.rados.rados.0.smithi190.stdout:320: finishing write tid 4 to smithi19039307-16 2023-11-01T04:17:10.679 INFO:tasks.rados.rados.0.smithi190.stdout:320: finishing write tid 5 to smithi19039307-16 2023-11-01T04:17:10.694 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:17:10.695 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-11-01T04:17:10.697 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:10.696+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:10.702 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:10.705 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:10.706 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-01T04:17:10.715 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:17:10.715 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-01T04:17:10.735 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:17:10.735 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-01T04:17:10.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~15s 2023-11-01T04:17:10.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~7s 2023-11-01T04:17:10.798 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:17:10.795+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:17:10.801 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:17:10.802 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-01T04:17:10.805 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:10.807 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:10.808 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-01T04:17:10.825 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:17:10.826 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-01T04:17:10.849 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:17:10.849 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-01T04:17:10.873 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:17:10.873 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-01T04:17:10.873 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-11-01T04:17:10.893 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:17:10.894 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-01T04:17:10.899 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:17:10.897+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:17:10.918 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:17:10.918 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-01T04:17:10.938 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:10.939 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:10.943 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:17:10.943 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-01T04:17:10.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-01T04:17:10.967 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:17:10.967 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-01T04:17:10.970 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.12 2023-11-01T04:17:10.980 INFO:tasks.rados.rados.0.smithi190.stdout:321: done (2 left) 2023-11-01T04:17:10.980 INFO:tasks.rados.rados.0.smithi190.stdout:322: done (1 left) 2023-11-01T04:17:10.980 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 44 current snap is 30 2023-11-01T04:17:10.981 INFO:tasks.rados.rados.0.smithi190.stdout:323: seq_num 119 ranges {0=458752,458752=507904,966656=507904,1474560=458752,1933312=81920} 2023-11-01T04:17:10.988 INFO:tasks.rados.rados.0.smithi190.stdout:323: writing smithi19039307-44 from 0 to 458752 tid 1 2023-11-01T04:17:10.995 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:17:10.995 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-01T04:17:10.996 INFO:tasks.rados.rados.0.smithi190.stdout:323: writing smithi19039307-44 from 458752 to 966656 tid 2 2023-11-01T04:17:11.000 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:10.998+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:11.005 INFO:tasks.rados.rados.0.smithi190.stdout:323: writing smithi19039307-44 from 966656 to 1474560 tid 3 2023-11-01T04:17:11.012 INFO:tasks.rados.rados.0.smithi190.stdout:323: writing smithi19039307-44 from 1474560 to 1933312 tid 4 2023-11-01T04:17:11.014 INFO:tasks.rados.rados.0.smithi190.stdout:323: writing smithi19039307-44 from 1933312 to 2015232 tid 5 2023-11-01T04:17:11.014 INFO:tasks.rados.rados.0.smithi190.stdout:324: read oid 21 snap -1 2023-11-01T04:17:11.015 INFO:tasks.rados.rados.0.smithi190.stdout:324: expect (ObjNum 59 snap 5 seq_num 59) 2023-11-01T04:17:11.015 INFO:tasks.rados.rados.0.smithi190.stdout:325: read oid 37 snap -1 2023-11-01T04:17:11.015 INFO:tasks.rados.rados.0.smithi190.stdout:325: expect (ObjNum 87 snap 20 seq_num 87) 2023-11-01T04:17:11.015 INFO:tasks.rados.rados.0.smithi190.stdout:326: copy_from oid 13 from oid 7 current snap is 30 2023-11-01T04:17:11.015 INFO:tasks.rados.rados.0.smithi190.stdout:320: finishing write tid 6 to smithi19039307-16 2023-11-01T04:17:11.016 INFO:tasks.rados.rados.0.smithi190.stdout:320: oid 16 updating version 0 to 123 2023-11-01T04:17:11.016 INFO:tasks.rados.rados.0.smithi190.stdout:320: oid 16 version 123 is already newer than 121 2023-11-01T04:17:11.016 INFO:tasks.rados.rados.0.smithi190.stdout:320: oid 16 version 123 is already newer than 122 2023-11-01T04:17:11.016 INFO:tasks.rados.rados.0.smithi190.stdout:320: oid 16 updating version 123 to 124 2023-11-01T04:17:11.016 INFO:tasks.rados.rados.0.smithi190.stdout:320: oid 16 version 124 is already newer than 120 2023-11-01T04:17:11.017 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 16 v 124 (ObjNum 118 snap 29 seq_num 118) dirty exists 2023-11-01T04:17:11.017 INFO:tasks.rados.rados.0.smithi190.stdout:320: left oid 16 (ObjNum 118 snap 29 seq_num 118) 2023-11-01T04:17:11.017 INFO:tasks.rados.rados.0.smithi190.stdout:320: done (4 left) 2023-11-01T04:17:11.017 INFO:tasks.rados.rados.0.smithi190.stdout:327: snap_create 2023-11-01T04:17:11.018 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:17:11.018 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-01T04:17:11.039 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:17:11.040 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-01T04:17:11.046 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-01T04:17:11.046 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-01T04:17:11.077 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:11.077 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:11.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-01T04:17:11.138 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:17:11.136+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:17:11.200 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:17:11.198+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:17:11.217 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:11.217 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:11.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-01T04:17:11.233 INFO:tasks.rados.rados.0.smithi190.stdout:324: expect (ObjNum 59 snap 5 seq_num 59) 2023-11-01T04:17:11.302 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:17:11.299+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:17:11.372 INFO:tasks.rados.rados.0.smithi190.stdout:323: finishing write tid 1 to smithi19039307-44 2023-11-01T04:17:11.373 INFO:tasks.rados.rados.0.smithi190.stdout:323: finishing write tid 2 to smithi19039307-44 2023-11-01T04:17:11.373 INFO:tasks.rados.rados.0.smithi190.stdout:323: finishing write tid 3 to smithi19039307-44 2023-11-01T04:17:11.373 INFO:tasks.rados.rados.0.smithi190.stdout:326: finishing copy_from racing read to smithi19039307-13 2023-11-01T04:17:11.373 INFO:tasks.rados.rados.0.smithi190.stdout:325: expect (ObjNum 87 snap 20 seq_num 87) 2023-11-01T04:17:11.400 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:11.401 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:11.402 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:17:11.399+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:17:11.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-01T04:17:11.502 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:11.511 INFO:tasks.rados.rados.0.smithi190.stdout:326: finishing copy_from to smithi19039307-13 2023-11-01T04:17:11.512 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 13 v 14 (ObjNum 6 snap 0 seq_num 6) dirty exists 2023-11-01T04:17:11.531 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:11.532 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:11.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-01T04:17:11.603 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:11.664 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:11.665 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:11.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-01T04:17:11.726 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:17:11.724+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:17:11.761 INFO:tasks.rados.rados.0.smithi190.stdout:323: finishing write tid 4 to smithi19039307-44 2023-11-01T04:17:11.800 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:11.800 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:11.807 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:17:11.805+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:17:11.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-01T04:17:11.862 INFO:tasks.rados.rados.0.smithi190.stdout:323: finishing write tid 5 to smithi19039307-44 2023-11-01T04:17:11.862 INFO:tasks.rados.rados.0.smithi190.stdout:323: finishing write tid 6 to smithi19039307-44 2023-11-01T04:17:11.907 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:17:11.905+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:17:11.980 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:11.980 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:11.985 INFO:tasks.rados.rados.0.smithi190.stdout:324: done (4 left) 2023-11-01T04:17:11.985 INFO:tasks.rados.rados.0.smithi190.stdout:325: done (3 left) 2023-11-01T04:17:11.986 INFO:tasks.rados.rados.0.smithi190.stdout:326: done (2 left) 2023-11-01T04:17:11.986 INFO:tasks.rados.rados.0.smithi190.stdout:327: done (1 left) 2023-11-01T04:17:11.986 INFO:tasks.rados.rados.0.smithi190.stdout:328: read oid 42 snap -1 2023-11-01T04:17:11.986 INFO:tasks.rados.rados.0.smithi190.stdout:328: expect (ObjNum 89 snap 20 seq_num 89) 2023-11-01T04:17:11.986 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 47 current snap is 31 2023-11-01T04:17:11.987 INFO:tasks.rados.rados.0.smithi190.stdout:329: seq_num 120 ranges {114688=475136,589824=655360,1245184=622592,1867776=278528} 2023-11-01T04:17:11.987 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-01T04:17:11.995 INFO:tasks.rados.rados.0.smithi190.stdout:329: writing smithi19039307-47 from 114688 to 589824 tid 1 2023-11-01T04:17:11.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-01T04:17:12.006 INFO:tasks.rados.rados.0.smithi190.stdout:329: writing smithi19039307-47 from 589824 to 1245184 tid 2 2023-11-01T04:17:12.016 INFO:tasks.rados.rados.0.smithi190.stdout:329: writing smithi19039307-47 from 1245184 to 1867776 tid 3 2023-11-01T04:17:12.020 INFO:tasks.rados.rados.0.smithi190.stdout:329: writing smithi19039307-47 from 1867776 to 2146304 tid 4 2023-11-01T04:17:12.021 INFO:tasks.rados.rados.0.smithi190.stdout:330: setattr oid 4 current snap is 31 2023-11-01T04:17:12.024 INFO:tasks.rados.rados.0.smithi190.stdout:331: delete oid 33 current snap is 31 2023-11-01T04:17:12.032 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:17:12.031+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:17:12.106 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:12.122 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:12.123 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:12.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-01T04:17:12.167 INFO:tasks.rados.rados.0.smithi190.stdout:323: finishing write tid 7 to smithi19039307-44 2023-11-01T04:17:12.167 INFO:tasks.rados.rados.0.smithi190.stdout:323: oid 44 updating version 0 to 20 2023-11-01T04:17:12.168 INFO:tasks.rados.rados.0.smithi190.stdout:323: oid 44 version 20 is already newer than 17 2023-11-01T04:17:12.168 INFO:tasks.rados.rados.0.smithi190.stdout:323: oid 44 version 20 is already newer than 18 2023-11-01T04:17:12.168 INFO:tasks.rados.rados.0.smithi190.stdout:323: oid 44 updating version 20 to 22 2023-11-01T04:17:12.168 INFO:tasks.rados.rados.0.smithi190.stdout:323: oid 44 version 22 is already newer than 19 2023-11-01T04:17:12.168 INFO:tasks.rados.rados.0.smithi190.stdout:323: oid 44 version 22 is already newer than 21 2023-11-01T04:17:12.168 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 44 v 22 (ObjNum 119 snap 30 seq_num 119) dirty exists 2023-11-01T04:17:12.169 INFO:tasks.rados.rados.0.smithi190.stdout:323: left oid 44 (ObjNum 119 snap 30 seq_num 119) 2023-11-01T04:17:12.208 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:12.263 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:12.263 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:12.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-01T04:17:12.309 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:12.314 INFO:tasks.rados.rados.0.smithi190.stdout:329: finishing write tid 1 to smithi19039307-47 2023-11-01T04:17:12.314 INFO:tasks.rados.rados.0.smithi190.stdout:329: finishing write tid 2 to smithi19039307-47 2023-11-01T04:17:12.315 INFO:tasks.rados.rados.0.smithi190.stdout:329: finishing write tid 3 to smithi19039307-47 2023-11-01T04:17:12.336 INFO:tasks.rados.rados.0.smithi190.stdout:323: done (4 left) 2023-11-01T04:17:12.336 INFO:tasks.rados.rados.0.smithi190.stdout:331: done (3 left) 2023-11-01T04:17:12.336 INFO:tasks.rados.rados.0.smithi190.stdout:332: read oid 32 snap -1 2023-11-01T04:17:12.337 INFO:tasks.rados.rados.0.smithi190.stdout:332: expect (ObjNum 6 snap 0 seq_num 6) 2023-11-01T04:17:12.337 INFO:tasks.rados.rados.0.smithi190.stdout:333: read oid 20 snap 29 2023-11-01T04:17:12.337 INFO:tasks.rados.rados.0.smithi190.stdout:333: expect (ObjNum 110 snap 24 seq_num 110) 2023-11-01T04:17:12.337 INFO:tasks.rados.rados.0.smithi190.stdout:334: delete oid 49 current snap is 31 2023-11-01T04:17:12.400 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:12.400 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:12.411 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:17:12.407+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:17:12.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-01T04:17:12.437 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 4 v 73 (ObjNum 3 snap 0 seq_num 3) dirty exists 2023-11-01T04:17:12.512 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:12.510+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:12.536 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:12.537 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:12.545 INFO:tasks.rados.rados.0.smithi190.stdout:328: expect (ObjNum 89 snap 20 seq_num 89) 2023-11-01T04:17:12.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-01T04:17:12.613 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:17:12.608+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:17:12.648 INFO:tasks.rados.rados.0.smithi190.stdout:333: expect (ObjNum 110 snap 24 seq_num 110) 2023-11-01T04:17:12.688 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:12.688 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:12.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-01T04:17:12.713 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:17:12.708+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:17:12.771 INFO:tasks.rados.rados.0.smithi190.stdout:329: finishing write tid 4 to smithi19039307-47 2023-11-01T04:17:12.772 INFO:tasks.rados.rados.0.smithi190.stdout:329: finishing write tid 5 to smithi19039307-47 2023-11-01T04:17:12.772 INFO:tasks.rados.rados.0.smithi190.stdout:332: expect (ObjNum 6 snap 0 seq_num 6) 2023-11-01T04:17:12.774 INFO:tasks.rados.rados.0.smithi190.stdout:329: finishing write tid 6 to smithi19039307-47 2023-11-01T04:17:12.774 INFO:tasks.rados.rados.0.smithi190.stdout:329: oid 47 updating version 0 to 35 2023-11-01T04:17:12.775 INFO:tasks.rados.rados.0.smithi190.stdout:329: oid 47 version 35 is already newer than 34 2023-11-01T04:17:12.775 INFO:tasks.rados.rados.0.smithi190.stdout:329: oid 47 updating version 35 to 36 2023-11-01T04:17:12.775 INFO:tasks.rados.rados.0.smithi190.stdout:329: oid 47 version 36 is already newer than 32 2023-11-01T04:17:12.775 INFO:tasks.rados.rados.0.smithi190.stdout:329: oid 47 version 36 is already newer than 33 2023-11-01T04:17:12.775 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 47 v 36 (ObjNum 120 snap 31 seq_num 120) dirty exists 2023-11-01T04:17:12.776 INFO:tasks.rados.rados.0.smithi190.stdout:329: left oid 47 (ObjNum 120 snap 31 seq_num 120) 2023-11-01T04:17:12.815 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:17:12.811+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:17:12.826 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:12.826 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:12.840 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-01T04:17:12.844 INFO:tasks.rados.rados.0.smithi190.stdout:328: done (5 left) 2023-11-01T04:17:12.844 INFO:tasks.rados.rados.0.smithi190.stdout:329: done (4 left) 2023-11-01T04:17:12.845 INFO:tasks.rados.rados.0.smithi190.stdout:330: done (3 left) 2023-11-01T04:17:12.845 INFO:tasks.rados.rados.0.smithi190.stdout:332: done (2 left) 2023-11-01T04:17:12.845 INFO:tasks.rados.rados.0.smithi190.stdout:333: done (1 left) 2023-11-01T04:17:12.845 INFO:tasks.rados.rados.0.smithi190.stdout:334: done (0 left) 2023-11-01T04:17:12.845 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 39 current snap is 31 2023-11-01T04:17:12.846 INFO:tasks.rados.rados.0.smithi190.stdout:335: seq_num 123 ranges {0=524288,524288=573440,1097728=688128,1785856=294912} 2023-11-01T04:17:12.852 INFO:tasks.rados.rados.0.smithi190.stdout:335: writing smithi19039307-39 from 0 to 524288 tid 1 2023-11-01T04:17:12.862 INFO:tasks.rados.rados.0.smithi190.stdout:335: writing smithi19039307-39 from 524288 to 1097728 tid 2 2023-11-01T04:17:12.874 INFO:tasks.rados.rados.0.smithi190.stdout:335: writing smithi19039307-39 from 1097728 to 1785856 tid 3 2023-11-01T04:17:12.878 INFO:tasks.rados.rados.0.smithi190.stdout:335: writing smithi19039307-39 from 1785856 to 2080768 tid 4 2023-11-01T04:17:12.879 INFO:tasks.rados.rados.0.smithi190.stdout:336: delete oid 50 current snap is 31 2023-11-01T04:17:12.915 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:17:12.913+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:17:12.961 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:12.961 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:12.975 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-01T04:17:12.989 INFO:teuthology.orchestra.run.smithi153.stderr:noscrub is unset 2023-11-01T04:17:13.047 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-01T04:17:13.060 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:17:13.059+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:17:13.086 INFO:tasks.rados.rados.0.smithi190.stdout:336: done (1 left) 2023-11-01T04:17:13.086 INFO:tasks.rados.rados.0.smithi190.stdout:337: copy_from oid 28 from oid 45 current snap is 31 2023-11-01T04:17:13.087 INFO:tasks.rados.rados.0.smithi190.stdout:338: read oid 35 snap -1 2023-11-01T04:17:13.087 INFO:tasks.rados.rados.0.smithi190.stdout:338: expect (ObjNum 77 snap 13 seq_num 77) 2023-11-01T04:17:13.087 INFO:tasks.rados.rados.0.smithi190.stdout:339: read oid 22 snap -1 2023-11-01T04:17:13.087 INFO:tasks.rados.rados.0.smithi190.stdout:339: expect (ObjNum 21 snap 0 seq_num 21) 2023-11-01T04:17:13.087 INFO:tasks.rados.rados.0.smithi190.stdout:340: copy_from oid 7 from oid 20 current snap is 31 2023-11-01T04:17:13.087 INFO:tasks.rados.rados.0.smithi190.stdout:341: delete oid 42 current snap is 31 2023-11-01T04:17:13.091 INFO:tasks.rados.rados.0.smithi190.stdout:337: finishing copy_from to smithi19039307-28 2023-11-01T04:17:13.091 INFO:tasks.rados.rados.0.smithi190.stdout:337: got expected ENOENT (src dne) 2023-11-01T04:17:13.091 INFO:tasks.rados.rados.0.smithi190.stdout:337: finishing copy_from racing read to smithi19039307-28 2023-11-01T04:17:13.096 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:13.097 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:13.111 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-01T04:17:13.156 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:17:13.154+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:17:13.164 INFO:tasks.rados.rados.0.smithi190.stdout:335: finishing write tid 1 to smithi19039307-39 2023-11-01T04:17:13.188 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2023-11-01T04:17:13.218 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:17:13.216+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:17:13.229 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:13.230 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:13.243 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-01T04:17:13.287 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:13.290 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:13.291 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['config', 'set', 'filestore_inject_stall', '3'] 2023-11-01T04:17:13.292 INFO:tasks.rados.rados.0.smithi190.stdout:337: done (5 left) 2023-11-01T04:17:13.292 INFO:tasks.rados.rados.0.smithi190.stdout:341: done (4 left) 2023-11-01T04:17:13.293 INFO:tasks.rados.rados.0.smithi190.stdout:342: snap_create 2023-11-01T04:17:13.314 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:17:13.315 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-11-01T04:17:13.336 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:17:13.336 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-01T04:17:13.336 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:17:13.334+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:17:13.355 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:17:13.355 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-01T04:17:13.375 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:17:13.376 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-01T04:17:13.398 INFO:tasks.rados.rados.0.smithi190.stdout:338: expect (ObjNum 77 snap 13 seq_num 77) 2023-11-01T04:17:13.410 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:13.410 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:13.414 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:17:13.414 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-01T04:17:13.423 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:17:13.417+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:17:13.425 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-01T04:17:13.438 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:17:13.438 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-01T04:17:13.460 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:17:13.461 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-01T04:17:13.461 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-11-01T04:17:13.482 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:17:13.482 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-01T04:17:13.506 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:17:13.506 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-01T04:17:13.519 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:13.525 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:17:13.526 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-01T04:17:13.538 INFO:tasks.rados.rados.0.smithi190.stdout:340: finishing copy_from racing read to smithi19039307-7 2023-11-01T04:17:13.547 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:17:13.547 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-01T04:17:13.548 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.12 2023-11-01T04:17:13.568 INFO:tasks.rados.rados.0.smithi190.stdout:335: finishing write tid 2 to smithi19039307-39 2023-11-01T04:17:13.568 INFO:tasks.rados.rados.0.smithi190.stdout:335: finishing write tid 3 to smithi19039307-39 2023-11-01T04:17:13.569 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:17:13.569 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-01T04:17:13.590 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:17:13.591 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-01T04:17:13.597 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:17:13.597 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:17:13.597 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2023-11-01T04:17:13.597 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:17:13.611 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-01T04:17:13.614 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:17:13.614 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-01T04:17:13.619 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-01T04:17:13.619 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-01T04:17:13.656 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:13.654+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:13.694 INFO:tasks.rados.rados.0.smithi190.stdout:339: expect (ObjNum 21 snap 0 seq_num 21) 2023-11-01T04:17:13.721 INFO:tasks.rados.rados.0.smithi190.stdout:340: finishing copy_from to smithi19039307-7 2023-11-01T04:17:13.721 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 7 v 117 (ObjNum 110 snap 24 seq_num 110) dirty exists 2023-11-01T04:17:13.721 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:17:13.719+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:17:13.729 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:17:13.729 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:17:13.729 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2023-11-01T04:17:13.729 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2023-11-01T04:17:13.730 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:17:13.743 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-01T04:17:13.820 INFO:tasks.rados.rados.0.smithi190.stdout:335: finishing write tid 4 to smithi19039307-39 2023-11-01T04:17:13.821 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:13.877 INFO:tasks.rados.rados.0.smithi190.stdout:335: finishing write tid 5 to smithi19039307-39 2023-11-01T04:17:13.910 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:17:13.910 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2023-11-01T04:17:13.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2023-11-01T04:17:13.911 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2023-11-01T04:17:13.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.911 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:198 3.4s0 3:2f6e6eee:::smithi19039307-7:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:13.911 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.965004+0000", 2023-11-01T04:17:13.912 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 56.942286607, 2023-11-01T04:17:13.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.37915321499999999, 2023-11-01T04:17:13.912 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:13.912 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:13.912 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:13.913 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:13.913 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:13.913 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 198 2023-11-01T04:17:13.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.913 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:13.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:13.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965004+0000", 2023-11-01T04:17:13.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:13.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:13.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965004+0000", 2023-11-01T04:17:13.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:13.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:13.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965005+0000", 2023-11-01T04:17:13.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8599999999999997e-07 2023-11-01T04:17:13.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:13.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965011+0000", 2023-11-01T04:17:13.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1970000000000001e-06 2023-11-01T04:17:13.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:13.918 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965117+0000", 2023-11-01T04:17:13.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000105919 2023-11-01T04:17:13.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:13.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965150+0000", 2023-11-01T04:17:13.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2468999999999997e-05 2023-11-01T04:17:13.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:13.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.041140+0000", 2023-11-01T04:17:13.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.075989950000000001 2023-11-01T04:17:13.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:13.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.041378+0000", 2023-11-01T04:17:13.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023841499999999999 2023-11-01T04:17:13.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:13.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.042565+0000", 2023-11-01T04:17:13.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011867480000000001 2023-11-01T04:17:13.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:13.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.145152+0000", 2023-11-01T04:17:13.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102587356 2023-11-01T04:17:13.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:13.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.344048+0000", 2023-11-01T04:17:13.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19889615599999999 2023-11-01T04:17:13.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:13.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.344158+0000", 2023-11-01T04:17:13.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000109219 2023-11-01T04:17:13.925 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:13.925 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:13.925 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:13.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.926 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:369 3.4s0 3:22b97aa6:::smithi19039307-42:head [append 0~196608] snapc d=[d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:17:13.926 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.064665+0000", 2023-11-01T04:17:13.926 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.842626369999998, 2023-11-01T04:17:13.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40249887899999998, 2023-11-01T04:17:13.927 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:13.927 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:13.927 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:13.927 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:13.927 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:13.927 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 369 2023-11-01T04:17:13.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.928 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:13.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:13.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.064665+0000", 2023-11-01T04:17:13.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:13.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:13.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.064665+0000", 2023-11-01T04:17:13.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:13.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:13.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.064666+0000", 2023-11-01T04:17:13.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.64e-06 2023-11-01T04:17:13.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.931 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:13.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.065011+0000", 2023-11-01T04:17:13.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034451099999999998 2023-11-01T04:17:13.932 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:13.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.065036+0000", 2023-11-01T04:17:13.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5148999999999999e-05 2023-11-01T04:17:13.932 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.933 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:13.933 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165152+0000", 2023-11-01T04:17:13.933 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100115738 2023-11-01T04:17:13.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:13.934 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.167023+0000", 2023-11-01T04:17:13.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001871075 2023-11-01T04:17:13.934 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:13.935 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.167186+0000", 2023-11-01T04:17:13.935 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016346199999999999 2023-11-01T04:17:13.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.935 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.935 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:13.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.168223+0000", 2023-11-01T04:17:13.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010366119999999999 2023-11-01T04:17:13.936 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.936 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:13.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.270989+0000", 2023-11-01T04:17:13.937 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102765841 2023-11-01T04:17:13.937 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.937 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.937 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:13.937 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.466957+0000", 2023-11-01T04:17:13.937 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.195968693 2023-11-01T04:17:13.938 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.938 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:13.938 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.467163+0000", 2023-11-01T04:17:13.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020615799999999999 2023-11-01T04:17:13.939 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:13.939 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:13.939 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:13.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.939 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:370 3.4s0 3:22b97aa6:::smithi19039307-42:head [setxattr _header (56) in=7b] snapc d=[d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:17:13.940 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.165200+0000", 2023-11-01T04:17:13.940 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.742091303999999, 2023-11-01T04:17:13.940 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40210787999999997, 2023-11-01T04:17:13.940 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:13.940 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:13.940 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:13.941 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:13.941 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:13.941 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 370 2023-11-01T04:17:13.941 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.941 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:13.942 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.942 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:13.942 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165200+0000", 2023-11-01T04:17:13.942 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:13.942 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.943 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:13.943 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165200+0000", 2023-11-01T04:17:13.943 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:13.943 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.943 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.944 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:13.944 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165202+0000", 2023-11-01T04:17:13.944 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6000000000000001e-06 2023-11-01T04:17:13.944 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.944 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.944 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:13.945 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165215+0000", 2023-11-01T04:17:13.945 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2904e-05 2023-11-01T04:17:13.945 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.945 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.945 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:13.945 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165249+0000", 2023-11-01T04:17:13.946 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3655000000000002e-05 2023-11-01T04:17:13.946 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.946 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.946 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:13.946 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165268+0000", 2023-11-01T04:17:13.947 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8810000000000001e-05 2023-11-01T04:17:13.947 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.947 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.947 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:13.947 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.169297+0000", 2023-11-01T04:17:13.947 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040292280000000001 2023-11-01T04:17:13.948 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.948 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.948 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:13.948 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.169474+0000", 2023-11-01T04:17:13.948 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017749500000000001 2023-11-01T04:17:13.949 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.949 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.949 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:13.949 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.170331+0000", 2023-11-01T04:17:13.949 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00085666300000000002 2023-11-01T04:17:13.949 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.950 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.950 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:13.950 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.271064+0000", 2023-11-01T04:17:13.950 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10073283099999999 2023-11-01T04:17:13.951 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.951 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.951 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:13.951 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.567166+0000", 2023-11-01T04:17:13.951 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29610207100000002 2023-11-01T04:17:13.952 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.952 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.952 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:13.952 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.567307+0000", 2023-11-01T04:17:13.952 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014162300000000001 2023-11-01T04:17:13.953 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:13.953 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:13.953 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:13.953 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.953 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.954 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:371 3.4s0 3:22b97aa6:::smithi19039307-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:17:13.954 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.165296+0000", 2023-11-01T04:17:13.954 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.741995226, 2023-11-01T04:17:13.954 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60529993999999998, 2023-11-01T04:17:13.954 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:13.955 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:13.955 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:13.955 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:13.955 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:13.955 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 371 2023-11-01T04:17:13.956 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.956 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:13.956 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.956 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:13.956 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165296+0000", 2023-11-01T04:17:13.957 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:13.957 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.957 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.957 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:13.957 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165296+0000", 2023-11-01T04:17:13.957 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:13.958 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.958 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.958 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:13.958 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165297+0000", 2023-11-01T04:17:13.958 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4729999999999999e-06 2023-11-01T04:17:13.959 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.959 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.959 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:13.959 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165304+0000", 2023-11-01T04:17:13.959 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7730000000000002e-06 2023-11-01T04:17:13.960 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.960 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.960 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:13.960 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165338+0000", 2023-11-01T04:17:13.960 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.392e-05 2023-11-01T04:17:13.960 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.961 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.961 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:13.961 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.165353+0000", 2023-11-01T04:17:13.961 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4922e-05 2023-11-01T04:17:13.961 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.962 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.962 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:13.962 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.266752+0000", 2023-11-01T04:17:13.962 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101399533 2023-11-01T04:17:13.962 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.962 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.963 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:13.963 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.266921+0000", 2023-11-01T04:17:13.963 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016895199999999999 2023-11-01T04:17:13.963 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.963 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.964 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:13.964 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.568170+0000", 2023-11-01T04:17:13.964 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30124910799999999 2023-11-01T04:17:13.964 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.964 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.964 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:13.965 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.568436+0000", 2023-11-01T04:17:13.965 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00026584599999999998 2023-11-01T04:17:13.965 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.965 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.965 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:13.966 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.770596+0000", 2023-11-01T04:17:13.966 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20215941300000001 2023-11-01T04:17:13.966 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:13.966 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:13.966 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:13.966 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.967 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.967 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:404 3.4s0 3:22b97aa6:::smithi19039307-42:head [read 0~1900544,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2023-11-01T04:17:13.967 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:33.200147+0000", 2023-11-01T04:17:13.967 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 40.707143742, 2023-11-01T04:17:13.967 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40700329400000002, 2023-11-01T04:17:13.968 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:13.968 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:13.968 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:13.968 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:13.968 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:13.968 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 404 2023-11-01T04:17:13.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.969 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:13.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.969 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:13.969 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200147+0000", 2023-11-01T04:17:13.969 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:13.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.970 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.970 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:13.970 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200147+0000", 2023-11-01T04:17:13.970 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:13.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.971 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.971 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:13.971 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200149+0000", 2023-11-01T04:17:13.971 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.187e-06 2023-11-01T04:17:13.971 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.972 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.972 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:13.972 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200162+0000", 2023-11-01T04:17:13.972 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2943e-05 2023-11-01T04:17:13.972 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.972 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.973 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:13.973 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200189+0000", 2023-11-01T04:17:13.973 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7056e-05 2023-11-01T04:17:13.973 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.973 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.973 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:13.974 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.200209+0000", 2023-11-01T04:17:13.974 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9802000000000001e-05 2023-11-01T04:17:13.974 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.974 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.974 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:13.975 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.403941+0000", 2023-11-01T04:17:13.975 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20373233099999999 2023-11-01T04:17:13.975 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.975 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.975 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:13.975 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.404113+0000", 2023-11-01T04:17:13.976 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000171855 2023-11-01T04:17:13.976 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.976 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.976 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:13.976 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:33.607150+0000", 2023-11-01T04:17:13.976 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20303711999999999 2023-11-01T04:17:13.977 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:13.977 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:13.977 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:13.977 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.977 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.977 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:449 3.4s0 3:2f6e6eee:::smithi19039307-7:15 [read 0~114688,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2023-11-01T04:17:13.978 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:39.142570+0000", 2023-11-01T04:17:13.978 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.764720830000002, 2023-11-01T04:17:13.978 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41050975099999998, 2023-11-01T04:17:13.978 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:13.978 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:13.979 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:13.979 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:13.979 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:13.979 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 449 2023-11-01T04:17:13.979 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.979 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:13.980 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.980 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:13.980 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.142570+0000", 2023-11-01T04:17:13.980 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:13.980 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.981 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.981 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:13.981 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.142570+0000", 2023-11-01T04:17:13.981 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:13.981 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.981 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.982 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:13.982 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.142571+0000", 2023-11-01T04:17:13.982 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.097e-06 2023-11-01T04:17:13.982 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.982 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.983 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:13.983 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.142578+0000", 2023-11-01T04:17:13.983 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.046e-06 2023-11-01T04:17:13.983 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.983 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.983 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:13.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.142600+0000", 2023-11-01T04:17:13.984 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1676999999999999e-05 2023-11-01T04:17:13.984 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.984 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.984 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:13.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.142614+0000", 2023-11-01T04:17:13.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4479e-05 2023-11-01T04:17:13.985 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.985 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.985 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:13.985 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.142867+0000", 2023-11-01T04:17:13.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025300499999999999 2023-11-01T04:17:13.986 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.986 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.986 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:13.986 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.142885+0000", 2023-11-01T04:17:13.986 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7385999999999999e-05 2023-11-01T04:17:13.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.987 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:13.987 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.450338+0000", 2023-11-01T04:17:13.987 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.307453219 2023-11-01T04:17:13.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.988 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.988 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:13.988 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.450500+0000", 2023-11-01T04:17:13.988 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016191 2023-11-01T04:17:13.988 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.988 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.989 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:13.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:39.553080+0000", 2023-11-01T04:17:13.989 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102579932 2023-11-01T04:17:13.989 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:13.989 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:13.989 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:13.990 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.990 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.990 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:547 3.4s0 3:2c54b66d:::smithi19039307-30:head [append 0~753664] snapc 1e=[1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:13.990 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:47.808599+0000", 2023-11-01T04:17:13.990 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.098691976000001, 2023-11-01T04:17:13.991 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41812574499999999, 2023-11-01T04:17:13.991 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:13.991 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:13.991 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:13.991 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:13.992 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:13.992 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 547 2023-11-01T04:17:13.992 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.992 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:13.992 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.992 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:13.993 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.808599+0000", 2023-11-01T04:17:13.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:13.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.993 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:13.993 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.808599+0000", 2023-11-01T04:17:13.994 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:13.994 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.994 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.994 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:13.994 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.808602+0000", 2023-11-01T04:17:13.995 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9859999999999999e-06 2023-11-01T04:17:13.995 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.995 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.995 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:13.995 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.809960+0000", 2023-11-01T04:17:13.995 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013583950000000001 2023-11-01T04:17:13.996 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.996 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:13.996 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.809995+0000", 2023-11-01T04:17:13.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4452000000000002e-05 2023-11-01T04:17:13.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:13.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.910089+0000", 2023-11-01T04:17:13.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100094554 2023-11-01T04:17:13.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.998 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:13.998 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.910257+0000", 2023-11-01T04:17:13.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016809600000000001 2023-11-01T04:17:13.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:13.999 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:13.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:13.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.910547+0000", 2023-11-01T04:17:13.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000289552 2023-11-01T04:17:13.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.000 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.000 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.913558+0000", 2023-11-01T04:17:14.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030110029999999999 2023-11-01T04:17:14.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.001 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.001 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.922892+0000", 2023-11-01T04:17:14.001 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0093338620000000001 2023-11-01T04:17:14.001 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.001 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.002 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.002 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.226596+0000", 2023-11-01T04:17:14.002 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30370408399999999 2023-11-01T04:17:14.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.002 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.226725+0000", 2023-11-01T04:17:14.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000128761 2023-11-01T04:17:14.003 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.003 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.003 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.004 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.004 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.004 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:548 3.4s0 3:2c54b66d:::smithi19039307-30:head [append 0~720896] snapc 1e=[1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:14.004 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:47.910162+0000", 2023-11-01T04:17:14.004 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 25.997128750999998, 2023-11-01T04:17:14.005 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41704092300000001, 2023-11-01T04:17:14.005 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.005 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.005 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.005 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.005 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.006 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 548 2023-11-01T04:17:14.006 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.006 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.006 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.006 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.006 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.910162+0000", 2023-11-01T04:17:14.007 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.007 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.007 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.007 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.007 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.910162+0000", 2023-11-01T04:17:14.007 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.008 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.008 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.008 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.008 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.910165+0000", 2023-11-01T04:17:14.008 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8930000000000001e-06 2023-11-01T04:17:14.009 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.009 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.009 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.009 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911468+0000", 2023-11-01T04:17:14.009 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013033599999999999 2023-11-01T04:17:14.009 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.010 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.010 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.010 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911502+0000", 2023-11-01T04:17:14.010 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3108999999999998e-05 2023-11-01T04:17:14.010 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.011 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.011 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.011 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911523+0000", 2023-11-01T04:17:14.011 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1282999999999999e-05 2023-11-01T04:17:14.011 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.012 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.012 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.012 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.915890+0000", 2023-11-01T04:17:14.012 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004367541 2023-11-01T04:17:14.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.013 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.013 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.013 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.916205+0000", 2023-11-01T04:17:14.013 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00031428400000000002 2023-11-01T04:17:14.013 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.013 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.014 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.014 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.918707+0000", 2023-11-01T04:17:14.014 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025018929999999998 2023-11-01T04:17:14.014 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.014 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.014 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.015 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.025371+0000", 2023-11-01T04:17:14.015 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.106664498 2023-11-01T04:17:14.015 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.015 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.015 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.016 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.327079+0000", 2023-11-01T04:17:14.016 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30170833499999999 2023-11-01T04:17:14.016 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.016 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.016 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.016 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.327203+0000", 2023-11-01T04:17:14.017 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012372699999999999 2023-11-01T04:17:14.017 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.017 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.017 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.017 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.017 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.018 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:549 3.4s0 3:2c54b66d:::smithi19039307-30:head [append 0~49152] snapc 1e=[1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:14.018 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:47.911560+0000", 2023-11-01T04:17:14.018 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 25.995730913999999, 2023-11-01T04:17:14.018 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.415940221, 2023-11-01T04:17:14.018 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.019 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.019 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.019 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.019 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.019 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 549 2023-11-01T04:17:14.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.020 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.020 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.020 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.020 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911560+0000", 2023-11-01T04:17:14.021 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.021 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.021 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.021 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911560+0000", 2023-11-01T04:17:14.022 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.022 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.022 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.022 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.022 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911562+0000", 2023-11-01T04:17:14.022 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0600000000000002e-06 2023-11-01T04:17:14.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.023 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.023 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911676+0000", 2023-11-01T04:17:14.023 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114405 2023-11-01T04:17:14.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.024 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911698+0000", 2023-11-01T04:17:14.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1625999999999999e-05 2023-11-01T04:17:14.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.025 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.025 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.025 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911718+0000", 2023-11-01T04:17:14.025 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9706000000000001e-05 2023-11-01T04:17:14.025 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.026 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.026 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.026 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.920665+0000", 2023-11-01T04:17:14.026 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0089476859999999998 2023-11-01T04:17:14.026 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.026 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.027 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.027 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.920862+0000", 2023-11-01T04:17:14.027 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000196443 2023-11-01T04:17:14.027 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.027 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.028 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.028 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.921831+0000", 2023-11-01T04:17:14.028 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00096887500000000001 2023-11-01T04:17:14.028 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.028 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.025374+0000", 2023-11-01T04:17:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103543049 2023-11-01T04:17:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.327401+0000", 2023-11-01T04:17:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30202717899999998 2023-11-01T04:17:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.327500+0000", 2023-11-01T04:17:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9191999999999995e-05 2023-11-01T04:17:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:550 3.4s0 3:2c54b66d:::smithi19039307-30:head [setxattr _header (57) in=7b] snapc 1e=[1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:47.911747+0000", 2023-11-01T04:17:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 25.995543767000001, 2023-11-01T04:17:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41617588700000002, 2023-11-01T04:17:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.034 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 550 2023-11-01T04:17:14.034 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.034 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.034 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.034 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911747+0000", 2023-11-01T04:17:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911747+0000", 2023-11-01T04:17:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911749+0000", 2023-11-01T04:17:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7e-06 2023-11-01T04:17:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911756+0000", 2023-11-01T04:17:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5490000000000001e-06 2023-11-01T04:17:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911776+0000", 2023-11-01T04:17:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9150000000000001e-05 2023-11-01T04:17:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911791+0000", 2023-11-01T04:17:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5407e-05 2023-11-01T04:17:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.923195+0000", 2023-11-01T04:17:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011403583 2023-11-01T04:17:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.923372+0000", 2023-11-01T04:17:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000177207 2023-11-01T04:17:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.924202+0000", 2023-11-01T04:17:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00083057400000000005 2023-11-01T04:17:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.025376+0000", 2023-11-01T04:17:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101173993 2023-11-01T04:17:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.327778+0000", 2023-11-01T04:17:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30240209600000001 2023-11-01T04:17:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.327923+0000", 2023-11-01T04:17:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000144628 2023-11-01T04:17:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:551 3.4s0 3:2c54b66d:::smithi19039307-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:47.911849+0000", 2023-11-01T04:17:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 25.995441403000001, 2023-11-01T04:17:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.72034387799999999, 2023-11-01T04:17:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 551 2023-11-01T04:17:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911849+0000", 2023-11-01T04:17:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911849+0000", 2023-11-01T04:17:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911851+0000", 2023-11-01T04:17:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.477e-06 2023-11-01T04:17:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911858+0000", 2023-11-01T04:17:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.063e-06 2023-11-01T04:17:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911900+0000", 2023-11-01T04:17:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1505999999999999e-05 2023-11-01T04:17:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:47.911919+0000", 2023-11-01T04:17:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9550000000000001e-05 2023-11-01T04:17:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.024812+0000", 2023-11-01T04:17:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.112892837 2023-11-01T04:17:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.024988+0000", 2023-11-01T04:17:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017627699999999999 2023-11-01T04:17:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.328738+0000", 2023-11-01T04:17:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30374931799999999 2023-11-01T04:17:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.328956+0000", 2023-11-01T04:17:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021849900000000001 2023-11-01T04:17:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:48.632193+0000", 2023-11-01T04:17:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30323735099999999 2023-11-01T04:17:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:624 3.bs0 3:d1d3cdc7:::smithi19039307-26:head [append 0~0] snapc 27=[27,26,22,1b,19,18,12,11] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2023-11-01T04:17:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:54.789464+0000", 2023-11-01T04:17:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.117827396999999, 2023-11-01T04:17:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41123595400000001, 2023-11-01T04:17:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 624 2023-11-01T04:17:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789464+0000", 2023-11-01T04:17:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789464+0000", 2023-11-01T04:17:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789466+0000", 2023-11-01T04:17:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9399999999999998e-06 2023-11-01T04:17:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789479+0000", 2023-11-01T04:17:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2595999999999999e-05 2023-11-01T04:17:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789514+0000", 2023-11-01T04:17:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4845999999999998e-05 2023-11-01T04:17:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789531+0000", 2023-11-01T04:17:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6799999999999998e-05 2023-11-01T04:17:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789669+0000", 2023-11-01T04:17:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013866099999999999 2023-11-01T04:17:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2023-11-01T04:17:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789686+0000", 2023-11-01T04:17:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7056999999999999e-05 2023-11-01T04:17:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.891232+0000", 2023-11-01T04:17:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101545627 2023-11-01T04:17:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.891421+0000", 2023-11-01T04:17:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018915100000000001 2023-11-01T04:17:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.893726+0000", 2023-11-01T04:17:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023044739999999999 2023-11-01T04:17:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.999788+0000", 2023-11-01T04:17:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.106062225 2023-11-01T04:17:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.200471+0000", 2023-11-01T04:17:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.200682834 2023-11-01T04:17:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.200699+0000", 2023-11-01T04:17:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000228743 2023-11-01T04:17:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:625 3.bs0 3:d1d3cdc7:::smithi19039307-26:head [rollback 12] snapc 27=[27,26,22,1b,19,18,12,11] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2023-11-01T04:17:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:54.789569+0000", 2023-11-01T04:17:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.117721909, 2023-11-01T04:17:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.412308589, 2023-11-01T04:17:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 625 2023-11-01T04:17:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789569+0000", 2023-11-01T04:17:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789569+0000", 2023-11-01T04:17:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789570+0000", 2023-11-01T04:17:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0299999999999998e-07 2023-11-01T04:17:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789574+0000", 2023-11-01T04:17:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5070000000000002e-06 2023-11-01T04:17:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789609+0000", 2023-11-01T04:17:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5046000000000003e-05 2023-11-01T04:17:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789622+0000", 2023-11-01T04:17:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3040000000000001e-05 2023-11-01T04:17:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.889948+0000", 2023-11-01T04:17:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10032589 2023-11-01T04:17:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.889971+0000", 2023-11-01T04:17:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2402999999999999e-05 2023-11-01T04:17:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.995630+0000", 2023-11-01T04:17:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10565901599999999 2023-11-01T04:17:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.995814+0000", 2023-11-01T04:17:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018381699999999999 2023-11-01T04:17:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.996939+0000", 2023-11-01T04:17:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011253089999999999 2023-11-01T04:17:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.999791+0000", 2023-11-01T04:17:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028522349999999998 2023-11-01T04:17:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.201667+0000", 2023-11-01T04:17:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201875794 2023-11-01T04:17:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.201878+0000", 2023-11-01T04:17:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021072899999999999 2023-11-01T04:17:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:626 3.bs0 3:d1d3cdc7:::smithi19039307-26:head [append 0~0] snapc 27=[27,26,22,1b,19,18,12,11] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2023-11-01T04:17:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:54.789661+0000", 2023-11-01T04:17:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.117630297000002, 2023-11-01T04:17:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.51282840500000004, 2023-11-01T04:17:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 626 2023-11-01T04:17:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789661+0000", 2023-11-01T04:17:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789661+0000", 2023-11-01T04:17:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789661+0000", 2023-11-01T04:17:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3e-07 2023-11-01T04:17:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789665+0000", 2023-11-01T04:17:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8430000000000003e-06 2023-11-01T04:17:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789677+0000", 2023-11-01T04:17:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2057e-05 2023-11-01T04:17:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.789688+0000", 2023-11-01T04:17:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0526000000000001e-05 2023-11-01T04:17:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.890065+0000", 2023-11-01T04:17:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100377709 2023-11-01T04:17:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.890076+0000", 2023-11-01T04:17:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0669999999999999e-05 2023-11-01T04:17:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.102 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.997793+0000", 2023-11-01T04:17:14.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.107716457 2023-11-01T04:17:14.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.102 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.997961+0000", 2023-11-01T04:17:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000168064 2023-11-01T04:17:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.998972+0000", 2023-11-01T04:17:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001011184 2023-11-01T04:17:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.200758+0000", 2023-11-01T04:17:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201785986 2023-11-01T04:17:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.302355+0000", 2023-11-01T04:17:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10159710499999999 2023-11-01T04:17:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:55.302489+0000", 2023-11-01T04:17:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000134074 2023-11-01T04:17:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:714 3.4s0 3:2c54b66d:::smithi19039307-30:head [append 0~753664] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.518796+0000", 2023-11-01T04:17:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.388495115, 2023-11-01T04:17:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40318051900000001, 2023-11-01T04:17:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 714 2023-11-01T04:17:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.518796+0000", 2023-11-01T04:17:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.518796+0000", 2023-11-01T04:17:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.518798+0000", 2023-11-01T04:17:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9099999999999999e-06 2023-11-01T04:17:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.519984+0000", 2023-11-01T04:17:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011858280000000001 2023-11-01T04:17:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.520019+0000", 2023-11-01T04:17:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5592000000000001e-05 2023-11-01T04:17:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.520047+0000", 2023-11-01T04:17:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7915999999999998e-05 2023-11-01T04:17:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.520190+0000", 2023-11-01T04:17:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000142473 2023-11-01T04:17:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.520401+0000", 2023-11-01T04:17:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021150800000000001 2023-11-01T04:17:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.522144+0000", 2023-11-01T04:17:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017434130000000001 2023-11-01T04:17:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.528429+0000", 2023-11-01T04:17:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006284258 2023-11-01T04:17:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.921854+0000", 2023-11-01T04:17:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.39342484900000002 2023-11-01T04:17:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.921976+0000", 2023-11-01T04:17:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000122772 2023-11-01T04:17:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:715 3.4s0 3:2c54b66d:::smithi19039307-30:head [append 0~688128] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.520089+0000", 2023-11-01T04:17:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.387201787, 2023-11-01T04:17:14.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40421347800000001, 2023-11-01T04:17:14.122 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.122 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.122 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.122 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.122 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.123 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 715 2023-11-01T04:17:14.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.123 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.520089+0000", 2023-11-01T04:17:14.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.520089+0000", 2023-11-01T04:17:14.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.520091+0000", 2023-11-01T04:17:14.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8199999999999999e-06 2023-11-01T04:17:14.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.521225+0000", 2023-11-01T04:17:14.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001134003 2023-11-01T04:17:14.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.521257+0000", 2023-11-01T04:17:14.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2496000000000002e-05 2023-11-01T04:17:14.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.521279+0000", 2023-11-01T04:17:14.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1869e-05 2023-11-01T04:17:14.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.523440+0000", 2023-11-01T04:17:14.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002161086 2023-11-01T04:17:14.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.523762+0000", 2023-11-01T04:17:14.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000321155 2023-11-01T04:17:14.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.526059+0000", 2023-11-01T04:17:14.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022970780000000001 2023-11-01T04:17:14.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.529897+0000", 2023-11-01T04:17:14.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038381040000000002 2023-11-01T04:17:14.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.133 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.133 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.924231+0000", 2023-11-01T04:17:14.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.39433468999999999 2023-11-01T04:17:14.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.134 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.134 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.924303+0000", 2023-11-01T04:17:14.134 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1177e-05 2023-11-01T04:17:14.134 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.134 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.135 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.135 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.135 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:716 3.4s0 3:2c54b66d:::smithi19039307-30:head [append 0~475136] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:14.135 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.521315+0000", 2023-11-01T04:17:14.136 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.385976003, 2023-11-01T04:17:14.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60362909399999998, 2023-11-01T04:17:14.136 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.136 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.136 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.137 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.137 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.137 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 716 2023-11-01T04:17:14.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.137 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.521315+0000", 2023-11-01T04:17:14.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.521315+0000", 2023-11-01T04:17:14.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.521317+0000", 2023-11-01T04:17:14.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0099999999999998e-06 2023-11-01T04:17:14.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.522189+0000", 2023-11-01T04:17:14.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00087234300000000003 2023-11-01T04:17:14.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.522232+0000", 2023-11-01T04:17:14.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3242999999999999e-05 2023-11-01T04:17:14.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622335+0000", 2023-11-01T04:17:14.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10010261500000001 2023-11-01T04:17:14.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622538+0000", 2023-11-01T04:17:14.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020300799999999999 2023-11-01T04:17:14.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622766+0000", 2023-11-01T04:17:14.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022807099999999999 2023-11-01T04:17:14.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.624385+0000", 2023-11-01T04:17:14.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001619073 2023-11-01T04:17:14.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.627002+0000", 2023-11-01T04:17:14.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026171440000000001 2023-11-01T04:17:14.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.124869+0000", 2023-11-01T04:17:14.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.49786697400000002 2023-11-01T04:17:14.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.124944+0000", 2023-11-01T04:17:14.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4612999999999999e-05 2023-11-01T04:17:14.149 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.149 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.150 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.150 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:717 3.4s0 3:2c54b66d:::smithi19039307-30:head [setxattr _header (57) in=7b] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:14.150 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.622376+0000", 2023-11-01T04:17:14.151 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.284914992999999, 2023-11-01T04:17:14.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50279286700000003, 2023-11-01T04:17:14.151 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.151 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.151 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.152 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.152 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.152 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 717 2023-11-01T04:17:14.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.152 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622376+0000", 2023-11-01T04:17:14.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622376+0000", 2023-11-01T04:17:14.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622378+0000", 2023-11-01T04:17:14.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9400000000000001e-06 2023-11-01T04:17:14.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622388+0000", 2023-11-01T04:17:14.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9629999999999993e-06 2023-11-01T04:17:14.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622407+0000", 2023-11-01T04:17:14.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8692999999999999e-05 2023-11-01T04:17:14.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622417+0000", 2023-11-01T04:17:14.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0966e-05 2023-11-01T04:17:14.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.625681+0000", 2023-11-01T04:17:14.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032634399999999998 2023-11-01T04:17:14.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.625865+0000", 2023-11-01T04:17:14.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018365800000000001 2023-11-01T04:17:14.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.626617+0000", 2023-11-01T04:17:14.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00075261700000000002 2023-11-01T04:17:14.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.627981+0000", 2023-11-01T04:17:14.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001364117 2023-11-01T04:17:14.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.125056+0000", 2023-11-01T04:17:14.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.49707428199999998 2023-11-01T04:17:14.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.125169+0000", 2023-11-01T04:17:14.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011319099999999999 2023-11-01T04:17:14.163 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.164 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.164 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.164 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:718 3.4s0 3:2c54b66d:::smithi19039307-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:14.165 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.622459+0000", 2023-11-01T04:17:14.165 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.284831412000001, 2023-11-01T04:17:14.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.70913759799999998, 2023-11-01T04:17:14.165 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.165 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:14.165 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.166 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.166 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.166 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 718 2023-11-01T04:17:14.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.166 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622459+0000", 2023-11-01T04:17:14.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622459+0000", 2023-11-01T04:17:14.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622460+0000", 2023-11-01T04:17:14.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0299999999999997e-07 2023-11-01T04:17:14.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622465+0000", 2023-11-01T04:17:14.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5900000000000001e-06 2023-11-01T04:17:14.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622478+0000", 2023-11-01T04:17:14.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.295e-05 2023-11-01T04:17:14.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.622487+0000", 2023-11-01T04:17:14.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7190000000000007e-06 2023-11-01T04:17:14.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.628970+0000", 2023-11-01T04:17:14.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006483234 2023-11-01T04:17:14.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:14.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.629139+0000", 2023-11-01T04:17:14.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016887599999999999 2023-11-01T04:17:14.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.125755+0000", 2023-11-01T04:17:14.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.496615957 2023-11-01T04:17:14.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.125921+0000", 2023-11-01T04:17:14.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000165818 2023-11-01T04:17:14.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.331597+0000", 2023-11-01T04:17:14.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20567655100000001 2023-11-01T04:17:14.177 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.177 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.177 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.178 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:742 3.4s0 3:22b97aa6:::smithi19039307-42:head [copy-from ver 30] snapc 30=[30,2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2023-11-01T04:17:14.178 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:03.911862+0000", 2023-11-01T04:17:14.178 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.9954291410000007, 2023-11-01T04:17:14.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.43783851299999998, 2023-11-01T04:17:14.178 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.179 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.179 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.179 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.179 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.179 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 742 2023-11-01T04:17:14.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.180 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.911862+0000", 2023-11-01T04:17:14.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.911862+0000", 2023-11-01T04:17:14.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.911865+0000", 2023-11-01T04:17:14.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8679999999999999e-06 2023-11-01T04:17:14.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.911876+0000", 2023-11-01T04:17:14.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0903e-05 2023-11-01T04:17:14.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.911899+0000", 2023-11-01T04:17:14.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3924999999999999e-05 2023-11-01T04:17:14.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:03.911913+0000", 2023-11-01T04:17:14.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3716999999999999e-05 2023-11-01T04:17:14.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.126362+0000", 2023-11-01T04:17:14.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.21444924400000001 2023-11-01T04:17:14.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.126515+0000", 2023-11-01T04:17:14.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015307100000000001 2023-11-01T04:17:14.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.240082+0000", 2023-11-01T04:17:14.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.113566865 2023-11-01T04:17:14.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.244253+0000", 2023-11-01T04:17:14.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004171129 2023-11-01T04:17:14.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.349538+0000", 2023-11-01T04:17:14.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105284816 2023-11-01T04:17:14.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:04.349700+0000", 2023-11-01T04:17:14.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016197499999999999 2023-11-01T04:17:14.191 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.191 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.191 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.192 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.192 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.192 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:17:14.193 INFO:tasks.rados.rados.0.smithi190.stdout:338: done (4 left) 2023-11-01T04:17:14.194 INFO:tasks.rados.rados.0.smithi190.stdout:339: done (3 left) 2023-11-01T04:17:14.194 INFO:tasks.rados.rados.0.smithi190.stdout:340: done (2 left) 2023-11-01T04:17:14.194 INFO:tasks.rados.rados.0.smithi190.stdout:342: done (1 left) 2023-11-01T04:17:14.194 INFO:tasks.rados.rados.0.smithi190.stdout:343: delete oid 16 current snap is 32 2023-11-01T04:17:14.194 INFO:tasks.rados.rados.0.smithi190.stdout:335: finishing write tid 6 to smithi19039307-39 2023-11-01T04:17:14.195 INFO:tasks.rados.rados.0.smithi190.stdout:335: oid 39 updating version 0 to 67 2023-11-01T04:17:14.195 INFO:tasks.rados.rados.0.smithi190.stdout:335: oid 39 version 67 is already newer than 65 2023-11-01T04:17:14.195 INFO:tasks.rados.rados.0.smithi190.stdout:335: oid 39 updating version 67 to 68 2023-11-01T04:17:14.195 INFO:tasks.rados.rados.0.smithi190.stdout:335: oid 39 version 68 is already newer than 66 2023-11-01T04:17:14.195 INFO:tasks.rados.rados.0.smithi190.stdout:335: oid 39 updating version 68 to 69 2023-11-01T04:17:14.196 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 39 v 69 (ObjNum 123 snap 31 seq_num 123) dirty exists 2023-11-01T04:17:14.196 INFO:tasks.rados.rados.0.smithi190.stdout:335: left oid 39 (ObjNum 123 snap 31 seq_num 123) 2023-11-01T04:17:14.197 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-01T04:17:14.197 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:17:14.190+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:17:14.294 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:17:14.292+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:17:14.329 INFO:tasks.rados.rados.0.smithi190.stdout:335: done (1 left) 2023-11-01T04:17:14.329 INFO:tasks.rados.rados.0.smithi190.stdout:343: done (0 left) 2023-11-01T04:17:14.330 INFO:tasks.rados.rados.0.smithi190.stdout:344: delete oid 8 current snap is 32 2023-11-01T04:17:14.370 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:17:14.370 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:17:14.371 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2023-11-01T04:17:14.371 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:17:14.384 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-01T04:17:14.394 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:14.495 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:14.505 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:17:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:17:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2023-11-01T04:17:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2023-11-01T04:17:14.506 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:17:14.519 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-01T04:17:14.539 INFO:tasks.rados.rados.0.smithi190.stdout:344: done (0 left) 2023-11-01T04:17:14.539 INFO:tasks.rados.rados.0.smithi190.stdout:345: delete oid 49 current snap is 32 2023-11-01T04:17:14.598 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:14.596+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:14.646 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:17:14.646 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2023-11-01T04:17:14.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2023-11-01T04:17:14.646 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2023-11-01T04:17:14.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.647 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(osd.13.22:1 3.6s0 3:6b2f2983:::smithi19039307-44:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:14.647 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.940531+0000", 2023-11-01T04:17:14.647 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 57.702009570000001, 2023-11-01T04:17:14.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30767406400000002, 2023-11-01T04:17:14.647 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.648 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:14.648 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940531+0000", 2023-11-01T04:17:14.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940531+0000", 2023-11-01T04:17:14.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940532+0000", 2023-11-01T04:17:14.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.86e-06 2023-11-01T04:17:14.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940541+0000", 2023-11-01T04:17:14.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7260000000000008e-06 2023-11-01T04:17:14.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940560+0000", 2023-11-01T04:17:14.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8647e-05 2023-11-01T04:17:14.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940571+0000", 2023-11-01T04:17:14.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1688999999999999e-05 2023-11-01T04:17:14.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.655 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940733+0000", 2023-11-01T04:17:14.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016122700000000001 2023-11-01T04:17:14.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2023-11-01T04:17:14.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.940754+0000", 2023-11-01T04:17:14.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.16e-05 2023-11-01T04:17:14.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.943338+0000", 2023-11-01T04:17:14.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025839080000000002 2023-11-01T04:17:14.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.943489+0000", 2023-11-01T04:17:14.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151268 2023-11-01T04:17:14.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.248205+0000", 2023-11-01T04:17:14.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30471513900000002 2023-11-01T04:17:14.660 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.660 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.660 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.661 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:201 3.7s0 3:ec8ceea5:::smithi19039307-25:2 [read 0~409600,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:14.661 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:16.965129+0000", 2023-11-01T04:17:14.661 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 57.677411292999999, 2023-11-01T04:17:14.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30806539799999999, 2023-11-01T04:17:14.661 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.662 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:14.662 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.662 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.662 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.662 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 201 2023-11-01T04:17:14.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.663 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965129+0000", 2023-11-01T04:17:14.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965129+0000", 2023-11-01T04:17:14.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965130+0000", 2023-11-01T04:17:14.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.387e-06 2023-11-01T04:17:14.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965138+0000", 2023-11-01T04:17:14.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8189999999999992e-06 2023-11-01T04:17:14.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965177+0000", 2023-11-01T04:17:14.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8946000000000003e-05 2023-11-01T04:17:14.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:16.965193+0000", 2023-11-01T04:17:14.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5739e-05 2023-11-01T04:17:14.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.067121+0000", 2023-11-01T04:17:14.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101928121 2023-11-01T04:17:14.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.067291+0000", 2023-11-01T04:17:14.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016974000000000001 2023-11-01T04:17:14.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.273194+0000", 2023-11-01T04:17:14.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.205903646 2023-11-01T04:17:14.672 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.672 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.672 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.673 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:214 3.7s0 3:e43a5132:::smithi19039307-40:head [append 0~0] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2023-11-01T04:17:14.673 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.953956+0000", 2023-11-01T04:17:14.673 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 56.688584339999998, 2023-11-01T04:17:14.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.310752374, 2023-11-01T04:17:14.673 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.674 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.674 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.674 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.674 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.674 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 214 2023-11-01T04:17:14.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.675 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.953956+0000", 2023-11-01T04:17:14.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.953956+0000", 2023-11-01T04:17:14.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.953959+0000", 2023-11-01T04:17:14.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2890000000000002e-06 2023-11-01T04:17:14.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.953978+0000", 2023-11-01T04:17:14.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8519999999999999e-05 2023-11-01T04:17:14.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954023+0000", 2023-11-01T04:17:14.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5640999999999999e-05 2023-11-01T04:17:14.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954048+0000", 2023-11-01T04:17:14.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4569000000000001e-05 2023-11-01T04:17:14.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954206+0000", 2023-11-01T04:17:14.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015823200000000001 2023-11-01T04:17:14.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2023-11-01T04:17:14.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954219+0000", 2023-11-01T04:17:14.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3052e-05 2023-11-01T04:17:14.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.055038+0000", 2023-11-01T04:17:14.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10081902299999999 2023-11-01T04:17:14.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.055200+0000", 2023-11-01T04:17:14.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000162304 2023-11-01T04:17:14.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.056576+0000", 2023-11-01T04:17:14.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013755169999999999 2023-11-01T04:17:14.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063225+0000", 2023-11-01T04:17:14.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066492729999999998 2023-11-01T04:17:14.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.264608+0000", 2023-11-01T04:17:14.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201382965 2023-11-01T04:17:14.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.264708+0000", 2023-11-01T04:17:14.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9988999999999994e-05 2023-11-01T04:17:14.689 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.689 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.689 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.690 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:215 3.7s0 3:e43a5132:::smithi19039307-40:head [rollback 3] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2023-11-01T04:17:14.690 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:17.954091+0000", 2023-11-01T04:17:14.690 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 56.688448964999999, 2023-11-01T04:17:14.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.31100268599999997, 2023-11-01T04:17:14.690 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.691 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.691 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.691 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.691 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.691 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 215 2023-11-01T04:17:14.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.692 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954091+0000", 2023-11-01T04:17:14.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954091+0000", 2023-11-01T04:17:14.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954092+0000", 2023-11-01T04:17:14.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.33e-07 2023-11-01T04:17:14.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954097+0000", 2023-11-01T04:17:14.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3770000000000004e-06 2023-11-01T04:17:14.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954111+0000", 2023-11-01T04:17:14.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3145999999999999e-05 2023-11-01T04:17:14.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954123+0000", 2023-11-01T04:17:14.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2296000000000001e-05 2023-11-01T04:17:14.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954349+0000", 2023-11-01T04:17:14.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022646499999999999 2023-11-01T04:17:14.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:14.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:17.954362+0000", 2023-11-01T04:17:14.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.312e-05 2023-11-01T04:17:14.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.057871+0000", 2023-11-01T04:17:14.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10350849299999999 2023-11-01T04:17:14.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.058109+0000", 2023-11-01T04:17:14.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000237631 2023-11-01T04:17:14.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.059576+0000", 2023-11-01T04:17:14.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001467154 2023-11-01T04:17:14.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.063386+0000", 2023-11-01T04:17:14.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038099010000000001 2023-11-01T04:17:14.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.265007+0000", 2023-11-01T04:17:14.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20162109 2023-11-01T04:17:14.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:18.265094+0000", 2023-11-01T04:17:14.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.708e-05 2023-11-01T04:17:14.705 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.706 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.706 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.706 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:372 3.7s0 3:ec8ceea5:::smithi19039307-25:head [append 0~442368] snapc d=[d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:17:14.707 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.071141+0000", 2023-11-01T04:17:14.707 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 46.571399544000002, 2023-11-01T04:17:14.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.323732619, 2023-11-01T04:17:14.707 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.708 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.708 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.708 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.708 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.708 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 372 2023-11-01T04:17:14.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.709 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.071141+0000", 2023-11-01T04:17:14.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.071141+0000", 2023-11-01T04:17:14.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.071143+0000", 2023-11-01T04:17:14.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8669999999999998e-06 2023-11-01T04:17:14.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.071960+0000", 2023-11-01T04:17:14.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081667000000000003 2023-11-01T04:17:14.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.071994+0000", 2023-11-01T04:17:14.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4363000000000003e-05 2023-11-01T04:17:14.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.714 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.072011+0000", 2023-11-01T04:17:14.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6350000000000001e-05 2023-11-01T04:17:14.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.072142+0000", 2023-11-01T04:17:14.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013155900000000001 2023-11-01T04:17:14.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2023-11-01T04:17:14.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.072155+0000", 2023-11-01T04:17:14.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2393e-05 2023-11-01T04:17:14.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.181494+0000", 2023-11-01T04:17:14.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.109339383 2023-11-01T04:17:14.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.181655+0000", 2023-11-01T04:17:14.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016068500000000001 2023-11-01T04:17:14.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.183750+0000", 2023-11-01T04:17:14.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020950859999999999 2023-11-01T04:17:14.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.192077+0000", 2023-11-01T04:17:14.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0083268979999999992 2023-11-01T04:17:14.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.721 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.721 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.394749+0000", 2023-11-01T04:17:14.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.202671929 2023-11-01T04:17:14.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.394873+0000", 2023-11-01T04:17:14.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012443600000000001 2023-11-01T04:17:14.722 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.723 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.723 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.723 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:373 3.7s0 3:ec8ceea5:::smithi19039307-25:head [append 0~720896] snapc d=[d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:17:14.724 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.079379+0000", 2023-11-01T04:17:14.724 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 46.563161395000002, 2023-11-01T04:17:14.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.61637890200000001, 2023-11-01T04:17:14.724 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.724 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.725 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.725 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.725 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.725 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 373 2023-11-01T04:17:14.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.726 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.079379+0000", 2023-11-01T04:17:14.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.079379+0000", 2023-11-01T04:17:14.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.079382+0000", 2023-11-01T04:17:14.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8830000000000002e-06 2023-11-01T04:17:14.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080658+0000", 2023-11-01T04:17:14.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001276461 2023-11-01T04:17:14.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080691+0000", 2023-11-01T04:17:14.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2793000000000003e-05 2023-11-01T04:17:14.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080707+0000", 2023-11-01T04:17:14.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6113e-05 2023-11-01T04:17:14.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.180945+0000", 2023-11-01T04:17:14.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100237957 2023-11-01T04:17:14.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:14.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.180960+0000", 2023-11-01T04:17:14.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5207e-05 2023-11-01T04:17:14.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.185900+0000", 2023-11-01T04:17:14.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049396860000000004 2023-11-01T04:17:14.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.186250+0000", 2023-11-01T04:17:14.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034973099999999999 2023-11-01T04:17:14.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.188479+0000", 2023-11-01T04:17:14.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022291809999999998 2023-11-01T04:17:14.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.192289+0000", 2023-11-01T04:17:14.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038101939999999998 2023-11-01T04:17:14.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.695644+0000", 2023-11-01T04:17:14.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50335487199999995 2023-11-01T04:17:14.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.739 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.739 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.695758+0000", 2023-11-01T04:17:14.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000113824 2023-11-01T04:17:14.739 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.739 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.740 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.740 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:374 3.7s0 3:ec8ceea5:::smithi19039307-25:head [append 0~49152] snapc d=[d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:17:14.740 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.080763+0000", 2023-11-01T04:17:14.741 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 46.561777360000001, 2023-11-01T04:17:14.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.61527303499999997, 2023-11-01T04:17:14.741 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.741 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.741 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.742 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.742 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.742 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 374 2023-11-01T04:17:14.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.742 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080763+0000", 2023-11-01T04:17:14.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080763+0000", 2023-11-01T04:17:14.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080764+0000", 2023-11-01T04:17:14.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7090000000000001e-06 2023-11-01T04:17:14.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080824+0000", 2023-11-01T04:17:14.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9429000000000003e-05 2023-11-01T04:17:14.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.747 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080839+0000", 2023-11-01T04:17:14.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5048999999999999e-05 2023-11-01T04:17:14.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080850+0000", 2023-11-01T04:17:14.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.097e-05 2023-11-01T04:17:14.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.181081+0000", 2023-11-01T04:17:14.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100230598 2023-11-01T04:17:14.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:14.750 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.181092+0000", 2023-11-01T04:17:14.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.152e-05 2023-11-01T04:17:14.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.750 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.751 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.190068+0000", 2023-11-01T04:17:14.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.008975841 2023-11-01T04:17:14.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.751 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.752 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.190248+0000", 2023-11-01T04:17:14.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000180318 2023-11-01T04:17:14.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.752 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.753 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.191149+0000", 2023-11-01T04:17:14.753 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00090082499999999995 2023-11-01T04:17:14.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.753 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.754 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.194083+0000", 2023-11-01T04:17:14.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029342859999999999 2023-11-01T04:17:14.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.754 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.754 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.754 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.695930+0000", 2023-11-01T04:17:14.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50184695999999995 2023-11-01T04:17:14.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.696036+0000", 2023-11-01T04:17:14.756 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010553 2023-11-01T04:17:14.756 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.756 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.756 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.757 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:375 3.7s0 3:ec8ceea5:::smithi19039307-25:head [setxattr _header (56) in=7b] snapc d=[d,c,b,4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:17:14.757 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.080873+0000", 2023-11-01T04:17:14.757 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 46.561667559999997, 2023-11-01T04:17:14.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.61552053200000001, 2023-11-01T04:17:14.758 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.758 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.758 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.758 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.759 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.759 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 375 2023-11-01T04:17:14.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.759 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080873+0000", 2023-11-01T04:17:14.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.761 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.761 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080873+0000", 2023-11-01T04:17:14.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080874+0000", 2023-11-01T04:17:14.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9999999999999995e-07 2023-11-01T04:17:14.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080879+0000", 2023-11-01T04:17:14.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5099999999999998e-06 2023-11-01T04:17:14.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080889+0000", 2023-11-01T04:17:14.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0176e-05 2023-11-01T04:17:14.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080901+0000", 2023-11-01T04:17:14.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1273e-05 2023-11-01T04:17:14.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.766 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.192640+0000", 2023-11-01T04:17:14.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.11173952600000001 2023-11-01T04:17:14.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.192811+0000", 2023-11-01T04:17:14.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017141500000000001 2023-11-01T04:17:14.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.193529+0000", 2023-11-01T04:17:14.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00071754999999999996 2023-11-01T04:17:14.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.194745+0000", 2023-11-01T04:17:14.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001215783 2023-11-01T04:17:14.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.696253+0000", 2023-11-01T04:17:14.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50150775000000003 2023-11-01T04:17:14.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.696393+0000", 2023-11-01T04:17:14.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014054900000000001 2023-11-01T04:17:14.771 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.771 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.772 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.772 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:376 3.7s0 3:ec8ceea5:::smithi19039307-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2023-11-01T04:17:14.772 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:28.080912+0000", 2023-11-01T04:17:14.773 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 46.561628501000001, 2023-11-01T04:17:14.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.91925732000000004, 2023-11-01T04:17:14.773 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.773 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:14.773 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.774 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.774 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.774 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 376 2023-11-01T04:17:14.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.775 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080912+0000", 2023-11-01T04:17:14.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080912+0000", 2023-11-01T04:17:14.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080912+0000", 2023-11-01T04:17:14.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9299999999999997e-07 2023-11-01T04:17:14.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080918+0000", 2023-11-01T04:17:14.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4229999999999996e-06 2023-11-01T04:17:14.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080931+0000", 2023-11-01T04:17:14.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3474e-05 2023-11-01T04:17:14.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.080940+0000", 2023-11-01T04:17:14.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2859999999999999e-06 2023-11-01T04:17:14.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.194154+0000", 2023-11-01T04:17:14.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.113214551 2023-11-01T04:17:14.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:14.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.194295+0000", 2023-11-01T04:17:14.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014040299999999999 2023-11-01T04:17:14.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.697223+0000", 2023-11-01T04:17:14.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50292827699999998 2023-11-01T04:17:14.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:28.697478+0000", 2023-11-01T04:17:14.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000255116 2023-11-01T04:17:14.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.786 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:29.000169+0000", 2023-11-01T04:17:14.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30269099700000002 2023-11-01T04:17:14.786 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.786 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.786 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.787 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:410 3.7s0 3:ec8ceea5:::smithi19039307-25:head [append 0~507904] snapc 12=[12,11,10,f,e,d,b,4] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-11-01T04:17:14.787 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:34.208836+0000", 2023-11-01T04:17:14.787 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 40.433704122000002, 2023-11-01T04:17:14.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41852388099999999, 2023-11-01T04:17:14.788 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.788 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.788 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.788 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.789 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.789 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 410 2023-11-01T04:17:14.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.789 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.790 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.790 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.208836+0000", 2023-11-01T04:17:14.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.790 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.790 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.208836+0000", 2023-11-01T04:17:14.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.208842+0000", 2023-11-01T04:17:14.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9499999999999998e-06 2023-11-01T04:17:14.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.210301+0000", 2023-11-01T04:17:14.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001459012 2023-11-01T04:17:14.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.210335+0000", 2023-11-01T04:17:14.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.392e-05 2023-11-01T04:17:14.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.210362+0000", 2023-11-01T04:17:14.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6971999999999999e-05 2023-11-01T04:17:14.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.311425+0000", 2023-11-01T04:17:14.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10106277299999999 2023-11-01T04:17:14.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.311594+0000", 2023-11-01T04:17:14.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016940299999999999 2023-11-01T04:17:14.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.313619+0000", 2023-11-01T04:17:14.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020251380000000001 2023-11-01T04:17:14.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.421318+0000", 2023-11-01T04:17:14.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10769884 2023-11-01T04:17:14.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.800 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.627230+0000", 2023-11-01T04:17:14.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20591240599999999 2023-11-01T04:17:14.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.627360+0000", 2023-11-01T04:17:14.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000129467 2023-11-01T04:17:14.801 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.802 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.802 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.802 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:411 3.7s0 3:ec8ceea5:::smithi19039307-25:head [append 0~507904] snapc 12=[12,11,10,f,e,d,b,4] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-11-01T04:17:14.803 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:34.210408+0000", 2023-11-01T04:17:14.803 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 40.432132132, 2023-11-01T04:17:14.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.51680641900000002, 2023-11-01T04:17:14.803 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.804 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.804 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.804 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.804 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.804 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 411 2023-11-01T04:17:14.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.805 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.210408+0000", 2023-11-01T04:17:14.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.210408+0000", 2023-11-01T04:17:14.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.210410+0000", 2023-11-01T04:17:14.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1799999999999999e-06 2023-11-01T04:17:14.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.211993+0000", 2023-11-01T04:17:14.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001582955 2023-11-01T04:17:14.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212047+0000", 2023-11-01T04:17:14.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4175999999999998e-05 2023-11-01T04:17:14.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212085+0000", 2023-11-01T04:17:14.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7969000000000002e-05 2023-11-01T04:17:14.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.417049+0000", 2023-11-01T04:17:14.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20496331200000001 2023-11-01T04:17:14.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.417369+0000", 2023-11-01T04:17:14.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032029900000000002 2023-11-01T04:17:14.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.419383+0000", 2023-11-01T04:17:14.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020145789999999998 2023-11-01T04:17:14.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.426157+0000", 2023-11-01T04:17:14.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0067739690000000003 2023-11-01T04:17:14.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.727103+0000", 2023-11-01T04:17:14.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30094544000000001 2023-11-01T04:17:14.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.727214+0000", 2023-11-01T04:17:14.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011154000000000001 2023-11-01T04:17:14.816 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.816 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.817 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.817 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:412 3.7s0 3:ec8ceea5:::smithi19039307-25:head [append 0~262144] snapc 12=[12,11,10,f,e,d,b,4] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-11-01T04:17:14.817 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:34.212131+0000", 2023-11-01T04:17:14.817 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 40.430408749999998, 2023-11-01T04:17:14.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.51556788200000003, 2023-11-01T04:17:14.818 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.818 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.818 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.818 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.819 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.819 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 412 2023-11-01T04:17:14.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.819 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212131+0000", 2023-11-01T04:17:14.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212131+0000", 2023-11-01T04:17:14.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212134+0000", 2023-11-01T04:17:14.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4930000000000001e-06 2023-11-01T04:17:14.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212623+0000", 2023-11-01T04:17:14.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00048915199999999997 2023-11-01T04:17:14.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212653+0000", 2023-11-01T04:17:14.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9703e-05 2023-11-01T04:17:14.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212675+0000", 2023-11-01T04:17:14.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2116e-05 2023-11-01T04:17:14.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.421675+0000", 2023-11-01T04:17:14.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.208999816 2023-11-01T04:17:14.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.421896+0000", 2023-11-01T04:17:14.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000221282 2023-11-01T04:17:14.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.423518+0000", 2023-11-01T04:17:14.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001622545 2023-11-01T04:17:14.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.426160+0000", 2023-11-01T04:17:14.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026410499999999998 2023-11-01T04:17:14.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.727590+0000", 2023-11-01T04:17:14.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30143093500000001 2023-11-01T04:17:14.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.727699+0000", 2023-11-01T04:17:14.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010878999999999999 2023-11-01T04:17:14.831 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.832 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.832 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.832 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:413 3.7s0 3:ec8ceea5:::smithi19039307-25:head [setxattr _header (57) in=7b] snapc 12=[12,11,10,f,e,d,b,4] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-11-01T04:17:14.833 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:34.212706+0000", 2023-11-01T04:17:14.833 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 40.42983426, 2023-11-01T04:17:14.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.51538549600000005, 2023-11-01T04:17:14.833 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.833 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.834 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.834 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.834 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.834 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 413 2023-11-01T04:17:14.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.835 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212706+0000", 2023-11-01T04:17:14.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212706+0000", 2023-11-01T04:17:14.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212708+0000", 2023-11-01T04:17:14.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.733e-06 2023-11-01T04:17:14.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212761+0000", 2023-11-01T04:17:14.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3742000000000001e-05 2023-11-01T04:17:14.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212785+0000", 2023-11-01T04:17:14.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3427000000000001e-05 2023-11-01T04:17:14.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.212801+0000", 2023-11-01T04:17:14.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5758999999999999e-05 2023-11-01T04:17:14.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.424813+0000", 2023-11-01T04:17:14.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.212012758 2023-11-01T04:17:14.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.424971+0000", 2023-11-01T04:17:14.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015783299999999999 2023-11-01T04:17:14.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.425686+0000", 2023-11-01T04:17:14.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00071460299999999996 2023-11-01T04:17:14.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.427284+0000", 2023-11-01T04:17:14.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001598152 2023-11-01T04:17:14.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.727935+0000", 2023-11-01T04:17:14.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30065153700000002 2023-11-01T04:17:14.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:34.728091+0000", 2023-11-01T04:17:14.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000155952 2023-11-01T04:17:14.847 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.847 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.847 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.848 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(osd.6.21:2 3.7s0 3:e030f458:::smithi19039307-11:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2023-11-01T04:17:14.848 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:49.295481+0000", 2023-11-01T04:17:14.848 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 25.347059051999999, 2023-11-01T04:17:14.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30315570400000003, 2023-11-01T04:17:14.848 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.849 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:14.849 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.295481+0000", 2023-11-01T04:17:14.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.295481+0000", 2023-11-01T04:17:14.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.295484+0000", 2023-11-01T04:17:14.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1130000000000001e-06 2023-11-01T04:17:14.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.295499+0000", 2023-11-01T04:17:14.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4619e-05 2023-11-01T04:17:14.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.295535+0000", 2023-11-01T04:17:14.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5833000000000001e-05 2023-11-01T04:17:14.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.395657+0000", 2023-11-01T04:17:14.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10012270500000001 2023-11-01T04:17:14.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.395812+0000", 2023-11-01T04:17:14.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015419699999999999 2023-11-01T04:17:14.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.395962+0000", 2023-11-01T04:17:14.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015025800000000001 2023-11-01T04:17:14.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:49.598637+0000", 2023-11-01T04:17:14.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20267497900000001 2023-11-01T04:17:14.858 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.858 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.858 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.859 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:590 3.7s0 3:ec8ceea5:::smithi19039307-25:head [rollback 22] snapc 22=[22,1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2023-11-01T04:17:14.859 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:50.148901+0000", 2023-11-01T04:17:14.859 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.493639419000001, 2023-11-01T04:17:14.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.31026156300000002, 2023-11-01T04:17:14.859 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.860 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.860 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.860 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.860 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.860 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 590 2023-11-01T04:17:14.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.861 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.148901+0000", 2023-11-01T04:17:14.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.148901+0000", 2023-11-01T04:17:14.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.148902+0000", 2023-11-01T04:17:14.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6000000000000002e-07 2023-11-01T04:17:14.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.148906+0000", 2023-11-01T04:17:14.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1130000000000003e-06 2023-11-01T04:17:14.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.148917+0000", 2023-11-01T04:17:14.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1253999999999999e-05 2023-11-01T04:17:14.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.148926+0000", 2023-11-01T04:17:14.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6330000000000002e-06 2023-11-01T04:17:14.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.152083+0000", 2023-11-01T04:17:14.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031574810000000002 2023-11-01T04:17:14.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.152393+0000", 2023-11-01T04:17:14.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00031033100000000001 2023-11-01T04:17:14.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.153380+0000", 2023-11-01T04:17:14.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00098682900000000005 2023-11-01T04:17:14.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.357809+0000", 2023-11-01T04:17:14.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.204429051 2023-11-01T04:17:14.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.458972+0000", 2023-11-01T04:17:14.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101162311 2023-11-01T04:17:14.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.459162+0000", 2023-11-01T04:17:14.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0001907 2023-11-01T04:17:14.872 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.873 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.873 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.873 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:591 3.7s0 3:ec8ceea5:::smithi19039307-25:head [append 0~0] snapc 22=[22,1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2023-11-01T04:17:14.874 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:50.148937+0000", 2023-11-01T04:17:14.874 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.493603057000001, 2023-11-01T04:17:14.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41081816900000001, 2023-11-01T04:17:14.874 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.874 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:14.874 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.875 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.875 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.875 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 591 2023-11-01T04:17:14.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.875 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.148937+0000", 2023-11-01T04:17:14.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.148937+0000", 2023-11-01T04:17:14.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.148938+0000", 2023-11-01T04:17:14.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4399999999999999e-07 2023-11-01T04:17:14.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.148943+0000", 2023-11-01T04:17:14.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3929999999999999e-06 2023-11-01T04:17:14.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.148956+0000", 2023-11-01T04:17:14.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2802999999999999e-05 2023-11-01T04:17:14.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.148964+0000", 2023-11-01T04:17:14.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.994e-06 2023-11-01T04:17:14.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.154591+0000", 2023-11-01T04:17:14.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056266340000000001 2023-11-01T04:17:14.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.154920+0000", 2023-11-01T04:17:14.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032974399999999998 2023-11-01T04:17:14.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2023-11-01T04:17:14.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.156292+0000", 2023-11-01T04:17:14.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013717180000000001 2023-11-01T04:17:14.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2023-11-01T04:17:14.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.357813+0000", 2023-11-01T04:17:14.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20152134299999999 2023-11-01T04:17:14.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2023-11-01T04:17:14.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.559578+0000", 2023-11-01T04:17:14.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20176448599999999 2023-11-01T04:17:14.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:50.559755+0000", 2023-11-01T04:17:14.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017730999999999999 2023-11-01T04:17:14.887 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.888 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.888 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.888 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 73/72 ECSubWriteReply(tid=109, last_complete=73'64, committed=1, applied=1))", 2023-11-01T04:17:14.889 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:16:54.242352+0000", 2023-11-01T04:17:14.889 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.400187823, 2023-11-01T04:17:14.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30607991699999998, 2023-11-01T04:17:14.889 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.889 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2023-11-01T04:17:14.890 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.242352+0000", 2023-11-01T04:17:14.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.242352+0000", 2023-11-01T04:17:14.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.242354+0000", 2023-11-01T04:17:14.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2729999999999999e-06 2023-11-01T04:17:14.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.242361+0000", 2023-11-01T04:17:14.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6899999999999992e-06 2023-11-01T04:17:14.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.242388+0000", 2023-11-01T04:17:14.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7199000000000002e-05 2023-11-01T04:17:14.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.242406+0000", 2023-11-01T04:17:14.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.753e-05 2023-11-01T04:17:14.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.545935+0000", 2023-11-01T04:17:14.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30352906699999999 2023-11-01T04:17:14.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:16:54.548432+0000", 2023-11-01T04:17:14.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024971580000000002 2023-11-01T04:17:14.897 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.898 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.898 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.899 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:774 3.7s0 3:e8826f72:::smithi19039307-5:head [read 0~1818624,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2023-11-01T04:17:14.899 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:08.964212+0000", 2023-11-01T04:17:14.899 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6783277999999999, 2023-11-01T04:17:14.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30254556300000002, 2023-11-01T04:17:14.899 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.900 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:14.900 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.900 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.900 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.901 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 774 2023-11-01T04:17:14.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.901 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964212+0000", 2023-11-01T04:17:14.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964212+0000", 2023-11-01T04:17:14.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964215+0000", 2023-11-01T04:17:14.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7460000000000001e-06 2023-11-01T04:17:14.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964229+0000", 2023-11-01T04:17:14.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3546999999999999e-05 2023-11-01T04:17:14.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964259+0000", 2023-11-01T04:17:14.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0886000000000002e-05 2023-11-01T04:17:14.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964276+0000", 2023-11-01T04:17:14.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6376e-05 2023-11-01T04:17:14.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.965751+0000", 2023-11-01T04:17:14.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014747969999999999 2023-11-01T04:17:14.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.966111+0000", 2023-11-01T04:17:14.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00036020099999999999 2023-11-01T04:17:14.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:09.266758+0000", 2023-11-01T04:17:14.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30064700999999999 2023-11-01T04:17:14.910 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.910 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.910 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.911 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:775 3.7s0 3:e8826f72:::smithi19039307-5:head [read 0~1818624,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2023-11-01T04:17:14.911 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:08.964308+0000", 2023-11-01T04:17:14.911 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6782324219999998, 2023-11-01T04:17:14.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30664408199999998, 2023-11-01T04:17:14.912 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.912 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:14.912 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.912 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.912 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.913 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 775 2023-11-01T04:17:14.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.913 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964308+0000", 2023-11-01T04:17:14.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964308+0000", 2023-11-01T04:17:14.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964309+0000", 2023-11-01T04:17:14.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0729999999999999e-06 2023-11-01T04:17:14.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964315+0000", 2023-11-01T04:17:14.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0870000000000003e-06 2023-11-01T04:17:14.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.918 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964327+0000", 2023-11-01T04:17:14.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1902000000000001e-05 2023-11-01T04:17:14.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964336+0000", 2023-11-01T04:17:14.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8030000000000003e-06 2023-11-01T04:17:14.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.967354+0000", 2023-11-01T04:17:14.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00301797 2023-11-01T04:17:14.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.967652+0000", 2023-11-01T04:17:14.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00029857199999999999 2023-11-01T04:17:14.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:09.270952+0000", 2023-11-01T04:17:14.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30329967499999999 2023-11-01T04:17:14.922 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.922 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.922 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.923 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4559.0:776 3.7s0 3:e8826f72:::smithi19039307-5:head [read 0~1818624,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2023-11-01T04:17:14.923 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2023-11-01T04:17:08.964387+0000", 2023-11-01T04:17:14.923 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6781531980000004, 2023-11-01T04:17:14.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40612100400000001, 2023-11-01T04:17:14.924 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2023-11-01T04:17:14.924 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2023-11-01T04:17:14.924 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2023-11-01T04:17:14.924 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4559", 2023-11-01T04:17:14.925 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:14.925 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 776 2023-11-01T04:17:14.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.925 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2023-11-01T04:17:14.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2023-11-01T04:17:14.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964387+0000", 2023-11-01T04:17:14.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.927 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2023-11-01T04:17:14.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964387+0000", 2023-11-01T04:17:14.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2023-11-01T04:17:14.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2023-11-01T04:17:14.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964388+0000", 2023-11-01T04:17:14.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5700000000000002e-07 2023-11-01T04:17:14.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2023-11-01T04:17:14.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964394+0000", 2023-11-01T04:17:14.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4330000000000001e-06 2023-11-01T04:17:14.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2023-11-01T04:17:14.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:08.964405+0000", 2023-11-01T04:17:14.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1229999999999999e-05 2023-11-01T04:17:14.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2023-11-01T04:17:14.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:09.064540+0000", 2023-11-01T04:17:14.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10013478100000001 2023-11-01T04:17:14.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.931 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2023-11-01T04:17:14.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:09.161565+0000", 2023-11-01T04:17:14.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.097024865000000002 2023-11-01T04:17:14.932 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2023-11-01T04:17:14.933 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:09.161916+0000", 2023-11-01T04:17:14.933 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00035119799999999999 2023-11-01T04:17:14.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2023-11-01T04:17:14.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2023-11-01T04:17:14.933 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2023-11-01T04:17:14.934 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2023-11-01T04:17:09.370508+0000", 2023-11-01T04:17:14.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20859174 2023-11-01T04:17:14.934 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.934 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.935 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.935 INFO:teuthology.orchestra.run.smithi181.stdout: } 2023-11-01T04:17:14.935 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-11-01T04:17:14.935 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:17:14.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-11-01T04:17:14.937 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:17:14.934+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:17:15.015 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-11-01T04:17:15.017 INFO:tasks.rados.rados.0.smithi190.stdout:345: done (0 left) 2023-11-01T04:17:15.018 INFO:tasks.rados.rados.0.smithi190.stdout:346: copy_from oid 33 from oid 9 current snap is 32 2023-11-01T04:17:15.018 INFO:tasks.rados.rados.0.smithi190.stdout:347: read oid 27 snap -1 2023-11-01T04:17:15.018 INFO:tasks.rados.rados.0.smithi190.stdout:347: expect (ObjNum 66 snap 6 seq_num 66) 2023-11-01T04:17:15.018 INFO:tasks.rados.rados.0.smithi190.stdout:348: rollback oid 42 current snap is 32 2023-11-01T04:17:15.019 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 42 to 23 2023-11-01T04:17:15.019 INFO:tasks.rados.rados.0.smithi190.stdout:349: read oid 29 snap -1 2023-11-01T04:17:15.019 INFO:tasks.rados.rados.0.smithi190.stdout:349: expect deleted 2023-11-01T04:17:15.019 INFO:tasks.rados.rados.0.smithi190.stdout:350: delete oid 19 current snap is 32 2023-11-01T04:17:15.024 INFO:tasks.rados.rados.0.smithi190.stdout:348: finishing rollback tid 1 to smithi19039307-42 2023-11-01T04:17:15.037 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:17:15.037+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:17:15.072 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:17:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2023-11-01T04:17:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(osd.1.22:2 3.0s0 3:044c8efd:::smithi19039307-9:head [assert-version v17,copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2023-11-01T04:17:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:15.021854+0000", 2023-11-01T04:17:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.050892737, 2023-11-01T04:17:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.050946274999999999, 2023-11-01T04:17:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.021854+0000", 2023-11-01T04:17:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.075 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.075 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.075 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.021854+0000", 2023-11-01T04:17:15.075 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.076 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.076 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.076 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.076 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.021857+0000", 2023-11-01T04:17:15.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2979999999999999e-06 2023-11-01T04:17:15.077 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.021864+0000", 2023-11-01T04:17:15.077 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5920000000000003e-06 2023-11-01T04:17:15.078 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.078 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.078 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.078 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.021879+0000", 2023-11-01T04:17:15.078 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5072000000000001e-05 2023-11-01T04:17:15.079 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.079 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.079 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.079 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.021891+0000", 2023-11-01T04:17:15.079 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1619e-05 2023-11-01T04:17:15.079 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.080 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.080 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.080 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.021999+0000", 2023-11-01T04:17:15.080 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010754199999999999 2023-11-01T04:17:15.080 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.081 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.081 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.081 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.022158+0000", 2023-11-01T04:17:15.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015985500000000001 2023-11-01T04:17:15.081 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.082 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.082 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.082 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpRead(3.0s0 93/78 ECSubRead(tid=98, to_read={3:044c8efd:::smithi19039307-9:head=0,409600,0}, subchunks={3:044c8efd:::smithi19039307-9:head=[0,1]}, attrs_to_read=))", 2023-11-01T04:17:15.083 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:15.022564+0000", 2023-11-01T04:17:15.083 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.050183669, 2023-11-01T04:17:15.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.050364049000000001, 2023-11-01T04:17:15.083 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.083 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:17:15.084 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.022564+0000", 2023-11-01T04:17:15.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.085 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.022564+0000", 2023-11-01T04:17:15.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.022564+0000", 2023-11-01T04:17:15.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.022564+0000", 2023-11-01T04:17:15.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.022590+0000", 2023-11-01T04:17:15.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6869e-05 2023-11-01T04:17:15.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.089 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.022608+0000", 2023-11-01T04:17:15.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7098000000000001e-05 2023-11-01T04:17:15.090 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.090 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.090 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.091 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 93/78 ECSubReadReply(tid=98, attrs_read=0))", 2023-11-01T04:17:15.091 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:15.024632+0000", 2023-11-01T04:17:15.091 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.048115195999999999, 2023-11-01T04:17:15.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.048372195999999999, 2023-11-01T04:17:15.091 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.092 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:17:15.092 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.024632+0000", 2023-11-01T04:17:15.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.024632+0000", 2023-11-01T04:17:15.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.024634+0000", 2023-11-01T04:17:15.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.559e-06 2023-11-01T04:17:15.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.025412+0000", 2023-11-01T04:17:15.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00077881499999999998 2023-11-01T04:17:15.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.025440+0000", 2023-11-01T04:17:15.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7219000000000001e-05 2023-11-01T04:17:15.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.025457+0000", 2023-11-01T04:17:15.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7346e-05 2023-11-01T04:17:15.098 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.098 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.098 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.099 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 93/78 ECSubReadReply(tid=98, attrs_read=0))", 2023-11-01T04:17:15.099 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:15.024937+0000", 2023-11-01T04:17:15.099 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.047809743000000002, 2023-11-01T04:17:15.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.048176488000000003, 2023-11-01T04:17:15.100 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.100 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:17:15.100 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.024937+0000", 2023-11-01T04:17:15.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.024937+0000", 2023-11-01T04:17:15.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.103 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.024938+0000", 2023-11-01T04:17:15.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4e-07 2023-11-01T04:17:15.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.026182+0000", 2023-11-01T04:17:15.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012440000000000001 2023-11-01T04:17:15.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.026195+0000", 2023-11-01T04:17:15.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2917e-05 2023-11-01T04:17:15.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.026203+0000", 2023-11-01T04:17:15.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0460000000000006e-06 2023-11-01T04:17:15.106 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.106 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.107 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.107 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.0s0 93/78 ECSubReadReply(tid=98, attrs_read=0))", 2023-11-01T04:17:15.107 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:15.025112+0000", 2023-11-01T04:17:15.108 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.047634930999999998, 2023-11-01T04:17:15.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.048069730999999997, 2023-11-01T04:17:15.108 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.108 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2023-11-01T04:17:15.109 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.025112+0000", 2023-11-01T04:17:15.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.025112+0000", 2023-11-01T04:17:15.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.025113+0000", 2023-11-01T04:17:15.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4000000000000001e-07 2023-11-01T04:17:15.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.112 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.112 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.026393+0000", 2023-11-01T04:17:15.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012795389999999999 2023-11-01T04:17:15.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.026414+0000", 2023-11-01T04:17:15.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1789000000000001e-05 2023-11-01T04:17:15.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.026426+0000", 2023-11-01T04:17:15.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1787000000000001e-05 2023-11-01T04:17:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2023-11-01T04:17:15.116 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 5 2023-11-01T04:17:15.116 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:17:15.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-11-01T04:17:15.128 INFO:tasks.rados.rados.0.smithi190.stdout:349: done (4 left) 2023-11-01T04:17:15.128 INFO:tasks.rados.rados.0.smithi190.stdout:350: done (3 left) 2023-11-01T04:17:15.128 INFO:tasks.rados.rados.0.smithi190.stdout:351: rollback oid 37 current snap is 32 2023-11-01T04:17:15.128 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 37 to 27 2023-11-01T04:17:15.129 INFO:tasks.rados.rados.0.smithi190.stdout:352: rollback oid 29 current snap is 32 2023-11-01T04:17:15.129 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 29 to 24 2023-11-01T04:17:15.129 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 22 current snap is 32 2023-11-01T04:17:15.129 INFO:tasks.rados.rados.0.smithi190.stdout:353: seq_num 130 ranges {360448=638976,999424=671744,1671168=802816,2473984=81920} 2023-11-01T04:17:15.138 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:15.134+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:15.143 INFO:tasks.rados.rados.0.smithi190.stdout:353: writing smithi19039307-22 from 360448 to 999424 tid 1 2023-11-01T04:17:15.154 INFO:tasks.rados.rados.0.smithi190.stdout:353: writing smithi19039307-22 from 999424 to 1671168 tid 2 2023-11-01T04:17:15.167 INFO:tasks.rados.rados.0.smithi190.stdout:353: writing smithi19039307-22 from 1671168 to 2473984 tid 3 2023-11-01T04:17:15.169 INFO:tasks.rados.rados.0.smithi190.stdout:353: writing smithi19039307-22 from 2473984 to 2555904 tid 4 2023-11-01T04:17:15.169 INFO:tasks.rados.rados.0.smithi190.stdout:354: read oid 48 snap -1 2023-11-01T04:17:15.170 INFO:tasks.rados.rados.0.smithi190.stdout:354: expect (ObjNum 6 snap 0 seq_num 6) 2023-11-01T04:17:15.170 INFO:tasks.rados.rados.0.smithi190.stdout:355: rollback oid 31 current snap is 32 2023-11-01T04:17:15.170 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 31 to 27 2023-11-01T04:17:15.170 INFO:tasks.rados.rados.0.smithi190.stdout:356: snap_create 2023-11-01T04:17:15.236 INFO:tasks.rados.rados.0.smithi190.stdout:351: finishing rollback tid 0 to smithi19039307-37 2023-11-01T04:17:15.236 INFO:tasks.rados.rados.0.smithi190.stdout:352: finishing rollback tid 1 to smithi19039307-29 2023-11-01T04:17:15.236 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 29 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2023-11-01T04:17:15.238 INFO:tasks.rados.rados.0.smithi190.stdout:346: finishing copy_from racing read to smithi19039307-33 2023-11-01T04:17:15.239 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:15.234+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:15.261 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:17:15.261 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:17:15.261 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2023-11-01T04:17:15.262 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2023-11-01T04:17:15.262 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:17:15.274 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-11-01T04:17:15.325 INFO:tasks.rados.rados.0.smithi190.stdout:348: finishing rollback tid 2 to smithi19039307-42 2023-11-01T04:17:15.326 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 42 v 119 (ObjNum 71 snap 8 seq_num 71) dirty exists 2023-11-01T04:17:15.339 INFO:tasks.rados.rados.0.smithi190.stdout:347: expect (ObjNum 66 snap 6 seq_num 66) 2023-11-01T04:17:15.339 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:17:15.337+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:17:15.439 INFO:tasks.rados.rados.0.smithi190.stdout:351: finishing rollback tid 1 to smithi19039307-37 2023-11-01T04:17:15.439 INFO:tasks.rados.rados.0.smithi190.stdout:346: finishing copy_from to smithi19039307-33 2023-11-01T04:17:15.440 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 33 v 17 (ObjNum 88 snap 20 seq_num 88) dirty exists 2023-11-01T04:17:15.440 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:15.436+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:15.443 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:17:15.443 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2023-11-01T04:17:15.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2023-11-01T04:17:15.444 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2023-11-01T04:17:15.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.444 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:138 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:17:15.444 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:15.283030+0000", 2023-11-01T04:17:15.444 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 60.157837104000002, 2023-11-01T04:17:15.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52972434099999999, 2023-11-01T04:17:15.445 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.445 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:15.445 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.445 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.446 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.446 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 138 2023-11-01T04:17:15.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.446 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283030+0000", 2023-11-01T04:17:15.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283030+0000", 2023-11-01T04:17:15.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283031+0000", 2023-11-01T04:17:15.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9600000000000008e-07 2023-11-01T04:17:15.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283035+0000", 2023-11-01T04:17:15.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7299999999999999e-06 2023-11-01T04:17:15.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283046+0000", 2023-11-01T04:17:15.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1249e-05 2023-11-01T04:17:15.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.283055+0000", 2023-11-01T04:17:15.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2660000000000007e-06 2023-11-01T04:17:15.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.290158+0000", 2023-11-01T04:17:15.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0071024260000000002 2023-11-01T04:17:15.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:15.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.290320+0000", 2023-11-01T04:17:15.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000161886 2023-11-01T04:17:15.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.610629+0000", 2023-11-01T04:17:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.32030918600000002 2023-11-01T04:17:15.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.610871+0000", 2023-11-01T04:17:15.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000242124 2023-11-01T04:17:15.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:15.812754+0000", 2023-11-01T04:17:15.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20188347800000001 2023-11-01T04:17:15.458 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.458 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.458 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.459 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:444 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~770048,checksum in=4b,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2023-11-01T04:17:15.459 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.143530+0000", 2023-11-01T04:17:15.459 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 36.297336819000002, 2023-11-01T04:17:15.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40681932599999998, 2023-11-01T04:17:15.459 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.460 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:15.460 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.460 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.460 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.461 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 444 2023-11-01T04:17:15.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.461 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.143530+0000", 2023-11-01T04:17:15.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.143530+0000", 2023-11-01T04:17:15.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.143534+0000", 2023-11-01T04:17:15.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2940000000000001e-06 2023-11-01T04:17:15.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.143566+0000", 2023-11-01T04:17:15.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2098999999999999e-05 2023-11-01T04:17:15.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.143602+0000", 2023-11-01T04:17:15.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6301999999999999e-05 2023-11-01T04:17:15.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243774+0000", 2023-11-01T04:17:15.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100171959 2023-11-01T04:17:15.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344071+0000", 2023-11-01T04:17:15.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100297211 2023-11-01T04:17:15.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2023-11-01T04:17:15.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344083+0000", 2023-11-01T04:17:15.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.203e-05 2023-11-01T04:17:15.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344705+0000", 2023-11-01T04:17:15.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000621515 2023-11-01T04:17:15.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344857+0000", 2023-11-01T04:17:15.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000152008 2023-11-01T04:17:15.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.550350+0000", 2023-11-01T04:17:15.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.205492908 2023-11-01T04:17:15.473 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.473 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.473 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.474 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:445 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~770048,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2023-11-01T04:17:15.474 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.243815+0000", 2023-11-01T04:17:15.474 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 36.197052466999999, 2023-11-01T04:17:15.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40799661599999998, 2023-11-01T04:17:15.474 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.475 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:15.475 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.475 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.475 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.475 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 445 2023-11-01T04:17:15.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.476 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243815+0000", 2023-11-01T04:17:15.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243815+0000", 2023-11-01T04:17:15.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.478 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243816+0000", 2023-11-01T04:17:15.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5999999999999999e-06 2023-11-01T04:17:15.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243826+0000", 2023-11-01T04:17:15.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4029999999999996e-06 2023-11-01T04:17:15.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243844+0000", 2023-11-01T04:17:15.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8076000000000001e-05 2023-11-01T04:17:15.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243855+0000", 2023-11-01T04:17:15.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.17e-05 2023-11-01T04:17:15.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344243+0000", 2023-11-01T04:17:15.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100387278 2023-11-01T04:17:15.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:15.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.344254+0000", 2023-11-01T04:17:15.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1e-05 2023-11-01T04:17:15.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.347410+0000", 2023-11-01T04:17:15.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031561829999999999 2023-11-01T04:17:15.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.347553+0000", 2023-11-01T04:17:15.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000142779 2023-11-01T04:17:15.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.651811+0000", 2023-11-01T04:17:15.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30425859700000002 2023-11-01T04:17:15.487 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.487 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.487 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.488 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:446 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~770048,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2023-11-01T04:17:15.488 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:39.243920+0000", 2023-11-01T04:17:15.488 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 36.196947283999997, 2023-11-01T04:17:15.488 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.408629468, 2023-11-01T04:17:15.489 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.489 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:15.489 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.489 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.489 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.489 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 446 2023-11-01T04:17:15.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.490 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243920+0000", 2023-11-01T04:17:15.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243920+0000", 2023-11-01T04:17:15.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243921+0000", 2023-11-01T04:17:15.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3399999999999998e-07 2023-11-01T04:17:15.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243926+0000", 2023-11-01T04:17:15.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.096e-06 2023-11-01T04:17:15.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243941+0000", 2023-11-01T04:17:15.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5569e-05 2023-11-01T04:17:15.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.495 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.243951+0000", 2023-11-01T04:17:15.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4499999999999993e-06 2023-11-01T04:17:15.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448384+0000", 2023-11-01T04:17:15.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20443328799999999 2023-11-01T04:17:15.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.448507+0000", 2023-11-01T04:17:15.498 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012274000000000001 2023-11-01T04:17:15.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:39.652549+0000", 2023-11-01T04:17:15.499 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20404249099999999 2023-11-01T04:17:15.499 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.499 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.499 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.500 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 77/72 ECSubWriteReply(tid=67, last_complete=77'54, committed=1, applied=1))", 2023-11-01T04:17:15.500 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:58.885525+0000", 2023-11-01T04:17:15.500 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 16.555342169999999, 2023-11-01T04:17:15.500 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40031678399999998, 2023-11-01T04:17:15.501 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.501 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:17:15.501 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:58.885525+0000", 2023-11-01T04:17:15.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:58.885525+0000", 2023-11-01T04:17:15.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:58.885527+0000", 2023-11-01T04:17:15.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.57e-06 2023-11-01T04:17:15.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.505 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:58.885537+0000", 2023-11-01T04:17:15.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0229e-05 2023-11-01T04:17:15.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:58.885566+0000", 2023-11-01T04:17:15.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8941e-05 2023-11-01T04:17:15.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.506 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:58.885585+0000", 2023-11-01T04:17:15.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9012999999999999e-05 2023-11-01T04:17:15.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:59.285792+0000", 2023-11-01T04:17:15.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40020705400000001 2023-11-01T04:17:15.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:59.285842+0000", 2023-11-01T04:17:15.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9976999999999999e-05 2023-11-01T04:17:15.509 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.509 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.509 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.510 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:684 3.8s0 3:1b945fb3:::smithi19039307-3:head [read 0~114688,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2023-11-01T04:17:15.510 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:02.265879+0000", 2023-11-01T04:17:15.511 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 13.174988579000001, 2023-11-01T04:17:15.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40463159399999998, 2023-11-01T04:17:15.511 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.511 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:15.511 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.512 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.512 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.512 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 684 2023-11-01T04:17:15.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.512 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:02.265879+0000", 2023-11-01T04:17:15.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:02.265879+0000", 2023-11-01T04:17:15.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:02.265880+0000", 2023-11-01T04:17:15.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.846e-06 2023-11-01T04:17:15.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:02.265888+0000", 2023-11-01T04:17:15.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8900000000000007e-06 2023-11-01T04:17:15.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:02.265910+0000", 2023-11-01T04:17:15.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2087999999999999e-05 2023-11-01T04:17:15.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.518 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.518 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.518 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:02.265926+0000", 2023-11-01T04:17:15.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5455999999999999e-05 2023-11-01T04:17:15.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.519 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:02.569013+0000", 2023-11-01T04:17:15.519 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30308672800000003 2023-11-01T04:17:15.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:02.569188+0000", 2023-11-01T04:17:15.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017518000000000001 2023-11-01T04:17:15.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:02.670510+0000", 2023-11-01T04:17:15.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101322406 2023-11-01T04:17:15.522 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.522 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.522 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.523 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:703 3.8s0 3:1514c4b8:::smithi19039307-15:head [append 0~360448] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:15.523 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:03.429968+0000", 2023-11-01T04:17:15.523 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 12.010899287999999, 2023-11-01T04:17:15.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.61723395999999997, 2023-11-01T04:17:15.523 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.524 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:15.524 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.524 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.524 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.524 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 703 2023-11-01T04:17:15.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.525 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.525 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.525 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.429968+0000", 2023-11-01T04:17:15.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.526 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.429968+0000", 2023-11-01T04:17:15.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.429970+0000", 2023-11-01T04:17:15.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3599999999999999e-06 2023-11-01T04:17:15.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.430545+0000", 2023-11-01T04:17:15.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00057518499999999998 2023-11-01T04:17:15.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.430578+0000", 2023-11-01T04:17:15.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2761e-05 2023-11-01T04:17:15.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.430602+0000", 2023-11-01T04:17:15.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4091999999999999e-05 2023-11-01T04:17:15.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.531870+0000", 2023-11-01T04:17:15.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10126766299999999 2023-11-01T04:17:15.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.531953+0000", 2023-11-01T04:17:15.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3191999999999999e-05 2023-11-01T04:17:15.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:15.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.532485+0000", 2023-11-01T04:17:15.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000532053 2023-11-01T04:17:15.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.535 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:15.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.535385+0000", 2023-11-01T04:17:15.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00289975 2023-11-01T04:17:15.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:15.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.046796+0000", 2023-11-01T04:17:15.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51141144299999997 2023-11-01T04:17:15.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.537 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.047202+0000", 2023-11-01T04:17:15.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00040546100000000003 2023-11-01T04:17:15.537 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.538 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.538 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.538 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:704 3.8s0 3:1514c4b8:::smithi19039307-15:head [setxattr _header (57) in=7b] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:15.539 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:03.430649+0000", 2023-11-01T04:17:15.539 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 12.010218216, 2023-11-01T04:17:15.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.61988206499999998, 2023-11-01T04:17:15.539 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.540 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:15.540 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.540 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.540 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.540 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 704 2023-11-01T04:17:15.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.541 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.430649+0000", 2023-11-01T04:17:15.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.430649+0000", 2023-11-01T04:17:15.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.430651+0000", 2023-11-01T04:17:15.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7999999999999999e-06 2023-11-01T04:17:15.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.430662+0000", 2023-11-01T04:17:15.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1420000000000001e-05 2023-11-01T04:17:15.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.430684+0000", 2023-11-01T04:17:15.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1804999999999999e-05 2023-11-01T04:17:15.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.546 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.430701+0000", 2023-11-01T04:17:15.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7456e-05 2023-11-01T04:17:15.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.534024+0000", 2023-11-01T04:17:15.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.103322419 2023-11-01T04:17:15.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.534202+0000", 2023-11-01T04:17:15.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000178653 2023-11-01T04:17:15.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.549 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:15.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.534937+0000", 2023-11-01T04:17:15.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00073435600000000003 2023-11-01T04:17:15.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:15.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.540299+0000", 2023-11-01T04:17:15.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0053623209999999998 2023-11-01T04:17:15.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:15.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.050137+0000", 2023-11-01T04:17:15.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.50983809700000005 2023-11-01T04:17:15.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.553 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.050531+0000", 2023-11-01T04:17:15.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039373799999999998 2023-11-01T04:17:15.553 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.553 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.553 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.554 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:705 3.8s0 3:1514c4b8:::smithi19039307-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:15.554 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:03.430726+0000", 2023-11-01T04:17:15.554 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 12.010141233000001, 2023-11-01T04:17:15.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.025818315, 2023-11-01T04:17:15.555 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.555 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:15.555 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.555 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.556 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.556 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 705 2023-11-01T04:17:15.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.556 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.430726+0000", 2023-11-01T04:17:15.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.430726+0000", 2023-11-01T04:17:15.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.430728+0000", 2023-11-01T04:17:15.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.843e-06 2023-11-01T04:17:15.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.430734+0000", 2023-11-01T04:17:15.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7699999999999998e-06 2023-11-01T04:17:15.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.430753+0000", 2023-11-01T04:17:15.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9789e-05 2023-11-01T04:17:15.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.430771+0000", 2023-11-01T04:17:15.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7682000000000001e-05 2023-11-01T04:17:15.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.535440+0000", 2023-11-01T04:17:15.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.104668866 2023-11-01T04:17:15.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:15.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.535500+0000", 2023-11-01T04:17:15.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9661999999999999e-05 2023-11-01T04:17:15.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.052600+0000", 2023-11-01T04:17:15.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.517100326 2023-11-01T04:17:15.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.052783+0000", 2023-11-01T04:17:15.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018343999999999999 2023-11-01T04:17:15.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.567 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.456544+0000", 2023-11-01T04:17:15.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40376093699999999 2023-11-01T04:17:15.567 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.568 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.568 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.568 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:725 3.8s0 3:1b945fb3:::smithi19039307-3:head [copy-from ver 64] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:15.569 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:03.470184+0000", 2023-11-01T04:17:15.569 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.970683382000001, 2023-11-01T04:17:15.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.68485487099999998, 2023-11-01T04:17:15.569 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.569 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:15.570 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.570 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.570 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.570 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 725 2023-11-01T04:17:15.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.571 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.470184+0000", 2023-11-01T04:17:15.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.470184+0000", 2023-11-01T04:17:15.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.470187+0000", 2023-11-01T04:17:15.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4599999999999999e-06 2023-11-01T04:17:15.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.470201+0000", 2023-11-01T04:17:15.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3789e-05 2023-11-01T04:17:15.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.470237+0000", 2023-11-01T04:17:15.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5871999999999998e-05 2023-11-01T04:17:15.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.470260+0000", 2023-11-01T04:17:15.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3564999999999999e-05 2023-11-01T04:17:15.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.535526+0000", 2023-11-01T04:17:15.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.065265104000000004 2023-11-01T04:17:15.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.535578+0000", 2023-11-01T04:17:15.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2685999999999997e-05 2023-11-01T04:17:15.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:15.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.943070+0000", 2023-11-01T04:17:15.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40749200400000002 2023-11-01T04:17:15.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:15.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.047232+0000", 2023-11-01T04:17:15.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.104162199 2023-11-01T04:17:15.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:15.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.154590+0000", 2023-11-01T04:17:15.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10735752 2023-11-01T04:17:15.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.155039+0000", 2023-11-01T04:17:15.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000448672 2023-11-01T04:17:15.583 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.583 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.583 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.583 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:726 3.8s0 3:1b945fb3:::smithi19039307-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:15.584 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:03.470307+0000", 2023-11-01T04:17:15.584 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.970560108000001, 2023-11-01T04:17:15.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.47587086899999997, 2023-11-01T04:17:15.584 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.584 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:15.585 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.585 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.585 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.585 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 726 2023-11-01T04:17:15.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.586 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.470307+0000", 2023-11-01T04:17:15.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.470307+0000", 2023-11-01T04:17:15.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.470309+0000", 2023-11-01T04:17:15.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8989999999999999e-06 2023-11-01T04:17:15.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.470317+0000", 2023-11-01T04:17:15.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3159999999999997e-06 2023-11-01T04:17:15.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.590 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.470338+0000", 2023-11-01T04:17:15.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0919000000000001e-05 2023-11-01T04:17:15.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.470356+0000", 2023-11-01T04:17:15.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7402999999999999e-05 2023-11-01T04:17:15.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.535734+0000", 2023-11-01T04:17:15.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.065378853000000001 2023-11-01T04:17:15.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2023-11-01T04:17:15.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.535817+0000", 2023-11-01T04:17:15.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3011999999999997e-05 2023-11-01T04:17:15.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.945454+0000", 2023-11-01T04:17:15.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40963697100000002 2023-11-01T04:17:15.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.945642+0000", 2023-11-01T04:17:15.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018746600000000001 2023-11-01T04:17:15.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.946178+0000", 2023-11-01T04:17:15.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00053602999999999995 2023-11-01T04:17:15.597 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.597 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.597 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.598 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:727 3.0s0 3:0e42a959:::smithi19039307-46:head [append 0~770048] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:15.598 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:03.494305+0000", 2023-11-01T04:17:15.598 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.94656178, 2023-11-01T04:17:15.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.5568381, 2023-11-01T04:17:15.599 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.599 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:15.599 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.599 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.599 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.600 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 727 2023-11-01T04:17:15.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.600 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.494305+0000", 2023-11-01T04:17:15.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.602 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.494305+0000", 2023-11-01T04:17:15.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.603 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.603 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.494308+0000", 2023-11-01T04:17:15.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1739999999999999e-06 2023-11-01T04:17:15.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.604 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.495115+0000", 2023-11-01T04:17:15.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00080778100000000004 2023-11-01T04:17:15.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.495148+0000", 2023-11-01T04:17:15.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2187999999999997e-05 2023-11-01T04:17:15.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.495163+0000", 2023-11-01T04:17:15.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5532999999999998e-05 2023-11-01T04:17:15.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.535699+0000", 2023-11-01T04:17:15.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.04053549 2023-11-01T04:17:15.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.608 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.608 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.535760+0000", 2023-11-01T04:17:15.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1783000000000001e-05 2023-11-01T04:17:15.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:15.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.536702+0000", 2023-11-01T04:17:15.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00094126799999999999 2023-11-01T04:17:15.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:15.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.540307+0000", 2023-11-01T04:17:15.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036051569999999999 2023-11-01T04:17:15.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:15.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.050752+0000", 2023-11-01T04:17:15.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51044498100000002 2023-11-01T04:17:15.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.051143+0000", 2023-11-01T04:17:15.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039174499999999998 2023-11-01T04:17:15.612 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.612 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.613 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.613 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:728 3.0s0 3:0e42a959:::smithi19039307-46:head [append 0~704512] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:15.613 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:03.499282+0000", 2023-11-01T04:17:15.614 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.941585113, 2023-11-01T04:17:15.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.55244297499999995, 2023-11-01T04:17:15.614 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.614 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:15.614 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.615 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.615 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.615 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 728 2023-11-01T04:17:15.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.615 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.499282+0000", 2023-11-01T04:17:15.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.499282+0000", 2023-11-01T04:17:15.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.499284+0000", 2023-11-01T04:17:15.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2060000000000001e-06 2023-11-01T04:17:15.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.500018+0000", 2023-11-01T04:17:15.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00073385900000000001 2023-11-01T04:17:15.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.500050+0000", 2023-11-01T04:17:15.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2248e-05 2023-11-01T04:17:15.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.500066+0000", 2023-11-01T04:17:15.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5418999999999999e-05 2023-11-01T04:17:15.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.537503+0000", 2023-11-01T04:17:15.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.037437353999999999 2023-11-01T04:17:15.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.537691+0000", 2023-11-01T04:17:15.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000187997 2023-11-01T04:17:15.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:15.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.539263+0000", 2023-11-01T04:17:15.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001571502 2023-11-01T04:17:15.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:15.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.541661+0000", 2023-11-01T04:17:15.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023983189999999999 2023-11-01T04:17:15.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:15.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.051353+0000", 2023-11-01T04:17:15.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.50969161399999996 2023-11-01T04:17:15.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.627 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.051725+0000", 2023-11-01T04:17:15.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00037245699999999997 2023-11-01T04:17:15.627 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.627 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.627 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.628 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:729 3.0s0 3:0e42a959:::smithi19039307-46:head [append 0~458752] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:15.628 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:03.502352+0000", 2023-11-01T04:17:15.628 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.938515324000001, 2023-11-01T04:17:15.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.54994150200000003, 2023-11-01T04:17:15.629 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.629 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:15.629 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.629 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.629 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.630 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 729 2023-11-01T04:17:15.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.630 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.502352+0000", 2023-11-01T04:17:15.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.502352+0000", 2023-11-01T04:17:15.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.502354+0000", 2023-11-01T04:17:15.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1959999999999998e-06 2023-11-01T04:17:15.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.502861+0000", 2023-11-01T04:17:15.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00050701800000000003 2023-11-01T04:17:15.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.502893+0000", 2023-11-01T04:17:15.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1899000000000001e-05 2023-11-01T04:17:15.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.502908+0000", 2023-11-01T04:17:15.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5421999999999999e-05 2023-11-01T04:17:15.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.540358+0000", 2023-11-01T04:17:15.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.037449457999999998 2023-11-01T04:17:15.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.540451+0000", 2023-11-01T04:17:15.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3382999999999997e-05 2023-11-01T04:17:15.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:15.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.541177+0000", 2023-11-01T04:17:15.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00072581200000000005 2023-11-01T04:17:15.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:15.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.542365+0000", 2023-11-01T04:17:15.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001188365 2023-11-01T04:17:15.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:15.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.051926+0000", 2023-11-01T04:17:15.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.50956085900000003 2023-11-01T04:17:15.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.052293+0000", 2023-11-01T04:17:15.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036708999999999998 2023-11-01T04:17:15.643 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.643 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.643 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.644 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:730 3.0s0 3:0e42a959:::smithi19039307-46:head [setxattr _header (57) in=7b] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:15.644 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:03.502936+0000", 2023-11-01T04:17:15.644 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.937930974, 2023-11-01T04:17:15.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.64751494300000001, 2023-11-01T04:17:15.644 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.645 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:15.645 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.645 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.645 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.645 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 730 2023-11-01T04:17:15.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.646 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.502936+0000", 2023-11-01T04:17:15.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.502936+0000", 2023-11-01T04:17:15.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.502937+0000", 2023-11-01T04:17:15.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.04e-06 2023-11-01T04:17:15.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.502943+0000", 2023-11-01T04:17:15.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6300000000000003e-06 2023-11-01T04:17:15.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.502954+0000", 2023-11-01T04:17:15.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1622e-05 2023-11-01T04:17:15.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.603104+0000", 2023-11-01T04:17:15.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100149423 2023-11-01T04:17:15.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.603279+0000", 2023-11-01T04:17:15.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017558 2023-11-01T04:17:15.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.603647+0000", 2023-11-01T04:17:15.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036773600000000002 2023-11-01T04:17:15.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:15.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.605074+0000", 2023-11-01T04:17:15.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014272009999999999 2023-11-01T04:17:15.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:15.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.606633+0000", 2023-11-01T04:17:15.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00155831 2023-11-01T04:17:15.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:15.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.150016+0000", 2023-11-01T04:17:15.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.543383387 2023-11-01T04:17:15.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.150451+0000", 2023-11-01T04:17:15.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043501400000000001 2023-11-01T04:17:15.658 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.658 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.659 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.659 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:731 3.0s0 3:0e42a959:::smithi19039307-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:15.659 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:03.603145+0000", 2023-11-01T04:17:15.660 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 11.837722150999999, 2023-11-01T04:17:15.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.156674121, 2023-11-01T04:17:15.660 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.660 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:15.660 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.661 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.661 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.661 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 731 2023-11-01T04:17:15.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.661 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.603145+0000", 2023-11-01T04:17:15.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.603145+0000", 2023-11-01T04:17:15.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.603147+0000", 2023-11-01T04:17:15.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6500000000000001e-06 2023-11-01T04:17:15.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.603155+0000", 2023-11-01T04:17:15.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1189999999999997e-06 2023-11-01T04:17:15.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.603174+0000", 2023-11-01T04:17:15.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9252000000000001e-05 2023-11-01T04:17:15.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.603186+0000", 2023-11-01T04:17:15.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2063e-05 2023-11-01T04:17:15.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.605837+0000", 2023-11-01T04:17:15.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026512229999999999 2023-11-01T04:17:15.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:15.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.606027+0000", 2023-11-01T04:17:15.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000189896 2023-11-01T04:17:15.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.657309+0000", 2023-11-01T04:17:15.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.051281331 2023-11-01T04:17:15.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.657503+0000", 2023-11-01T04:17:15.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019426999999999999 2023-11-01T04:17:15.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.759819+0000", 2023-11-01T04:17:15.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.102316317 2023-11-01T04:17:15.672 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.672 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.672 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.673 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:763 3.8s0 3:1514c4b8:::smithi19039307-15:head [read 0~1966080,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2023-11-01T04:17:15.673 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:08.964502+0000", 2023-11-01T04:17:15.673 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.4763650540000004, 2023-11-01T04:17:15.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.50875449699999997, 2023-11-01T04:17:15.674 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.674 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:15.674 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.674 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.674 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.675 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 763 2023-11-01T04:17:15.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.675 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964502+0000", 2023-11-01T04:17:15.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964502+0000", 2023-11-01T04:17:15.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964506+0000", 2023-11-01T04:17:15.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8199999999999998e-06 2023-11-01T04:17:15.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964522+0000", 2023-11-01T04:17:15.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6146000000000001e-05 2023-11-01T04:17:15.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964564+0000", 2023-11-01T04:17:15.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1541000000000001e-05 2023-11-01T04:17:15.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964589+0000", 2023-11-01T04:17:15.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5539000000000001e-05 2023-11-01T04:17:15.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964765+0000", 2023-11-01T04:17:15.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017567400000000001 2023-11-01T04:17:15.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.682 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2023-11-01T04:17:15.682 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964782+0000", 2023-11-01T04:17:15.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6838999999999999e-05 2023-11-01T04:17:15.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.070036+0000", 2023-11-01T04:17:15.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.105254654 2023-11-01T04:17:15.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.070229+0000", 2023-11-01T04:17:15.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000192571 2023-11-01T04:17:15.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.473257+0000", 2023-11-01T04:17:15.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40302771300000001 2023-11-01T04:17:15.685 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.686 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.686 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.686 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:764 3.8s0 3:1514c4b8:::smithi19039307-15:head [read 0~1966080,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2023-11-01T04:17:15.686 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:08.964648+0000", 2023-11-01T04:17:15.687 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.4762189809999997, 2023-11-01T04:17:15.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51097127799999997, 2023-11-01T04:17:15.687 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.687 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:15.687 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.688 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.688 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.688 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 764 2023-11-01T04:17:15.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.688 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964648+0000", 2023-11-01T04:17:15.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964648+0000", 2023-11-01T04:17:15.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.691 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.691 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964650+0000", 2023-11-01T04:17:15.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.407e-06 2023-11-01T04:17:15.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964658+0000", 2023-11-01T04:17:15.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.439e-06 2023-11-01T04:17:15.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964681+0000", 2023-11-01T04:17:15.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2772000000000002e-05 2023-11-01T04:17:15.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964693+0000", 2023-11-01T04:17:15.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1773000000000001e-05 2023-11-01T04:17:15.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964919+0000", 2023-11-01T04:17:15.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000226152 2023-11-01T04:17:15.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:15.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964932+0000", 2023-11-01T04:17:15.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3326000000000001e-05 2023-11-01T04:17:15.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.070992+0000", 2023-11-01T04:17:15.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.106060262 2023-11-01T04:17:15.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.071202+0000", 2023-11-01T04:17:15.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021012 2023-11-01T04:17:15.699 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.699 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.699 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.699 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.475619+0000", 2023-11-01T04:17:15.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40441702699999998 2023-11-01T04:17:15.700 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.700 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.700 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.700 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.701 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:765 3.8s0 3:1514c4b8:::smithi19039307-15:head [read 0~1966080,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2023-11-01T04:17:15.701 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:08.964718+0000", 2023-11-01T04:17:15.701 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.4761487080000002, 2023-11-01T04:17:15.701 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.613636089, 2023-11-01T04:17:15.702 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.702 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:15.702 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:15.702 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:15.702 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:15.703 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 765 2023-11-01T04:17:15.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.703 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964718+0000", 2023-11-01T04:17:15.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964718+0000", 2023-11-01T04:17:15.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964720+0000", 2023-11-01T04:17:15.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.176e-06 2023-11-01T04:17:15.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964727+0000", 2023-11-01T04:17:15.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4270000000000002e-06 2023-11-01T04:17:15.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964743+0000", 2023-11-01T04:17:15.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6242000000000001e-05 2023-11-01T04:17:15.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.708 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.708 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964758+0000", 2023-11-01T04:17:15.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.432e-05 2023-11-01T04:17:15.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964986+0000", 2023-11-01T04:17:15.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000227992 2023-11-01T04:17:15.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:15.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:08.964997+0000", 2023-11-01T04:17:15.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1493e-05 2023-11-01T04:17:15.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.074009+0000", 2023-11-01T04:17:15.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.109012368 2023-11-01T04:17:15.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:15.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.074177+0000", 2023-11-01T04:17:15.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000167138 2023-11-01T04:17:15.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.578355+0000", 2023-11-01T04:17:15.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.50417793300000002 2023-11-01T04:17:15.714 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.714 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.714 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.715 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpReadReply(3.8s0 88/78 ECSubReadReply(tid=88, attrs_read=0))", 2023-11-01T04:17:15.715 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:09.074284+0000", 2023-11-01T04:17:15.715 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.3665830239999996, 2023-11-01T04:17:15.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.401400908, 2023-11-01T04:17:15.716 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:15.716 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:17:15.716 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:15.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.716 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:15.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.074284+0000", 2023-11-01T04:17:15.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:15.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.074284+0000", 2023-11-01T04:17:15.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:15.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:15.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.074285+0000", 2023-11-01T04:17:15.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1939999999999999e-06 2023-11-01T04:17:15.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:15.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.075921+0000", 2023-11-01T04:17:15.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016353240000000001 2023-11-01T04:17:15.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:15.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.075952+0000", 2023-11-01T04:17:15.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0908999999999998e-05 2023-11-01T04:17:15.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:15.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.176053+0000", 2023-11-01T04:17:15.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100101914 2023-11-01T04:17:15.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:15.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.473538+0000", 2023-11-01T04:17:15.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29748490599999999 2023-11-01T04:17:15.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:15.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:15.723 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:15.723 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:09.475685+0000", 2023-11-01T04:17:15.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021466609999999998 2023-11-01T04:17:15.724 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.724 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.724 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.724 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:15.724 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:15.725 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:17:15.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-01T04:17:15.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-01T04:17:15.728 INFO:tasks.rados.rados.0.smithi190.stdout:355: finishing rollback tid 0 to smithi19039307-31 2023-11-01T04:17:15.728 INFO:tasks.rados.rados.0.smithi190.stdout:355: finishing rollback tid 1 to smithi19039307-31 2023-11-01T04:17:15.728 INFO:tasks.rados.rados.0.smithi190.stdout:355: finishing rollback tid 2 to smithi19039307-31 2023-11-01T04:17:15.729 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 31 v 77 (ObjNum 116 snap 25 seq_num 116) dirty exists 2023-11-01T04:17:15.729 INFO:tasks.rados.rados.0.smithi190.stdout:351: finishing rollback tid 2 to smithi19039307-37 2023-11-01T04:17:15.729 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 37 v 60 (ObjNum 87 snap 20 seq_num 87) dirty exists 2023-11-01T04:17:15.729 INFO:tasks.rados.rados.0.smithi190.stdout:353: finishing write tid 1 to smithi19039307-22 2023-11-01T04:17:15.729 INFO:tasks.rados.rados.0.smithi190.stdout:353: finishing write tid 2 to smithi19039307-22 2023-11-01T04:17:15.730 INFO:tasks.rados.rados.0.smithi190.stdout:353: finishing write tid 3 to smithi19039307-22 2023-11-01T04:17:15.730 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:17:15.723+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:17:15.826 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:15.884 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:15.887 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:15.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-01T04:17:15.889 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:17:15.889 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-11-01T04:17:15.909 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:17:15.910 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-01T04:17:15.930 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:17:15.930 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-01T04:17:15.932 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:17:15.933+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:17:15.946 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:17:15.947 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-01T04:17:15.954 INFO:tasks.rados.rados.0.smithi190.stdout:353: finishing write tid 4 to smithi19039307-22 2023-11-01T04:17:15.955 INFO:tasks.rados.rados.0.smithi190.stdout:353: finishing write tid 5 to smithi19039307-22 2023-11-01T04:17:15.972 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:17:15.972 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-01T04:17:15.994 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:17:15.995 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:15.995 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-01T04:17:15.997 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:15.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-01T04:17:16.015 INFO:teuthology.orchestra.run.smithi153.stderr:nodeep-scrub is unset 2023-11-01T04:17:16.016 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:17:16.016 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-01T04:17:16.019 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-11-01T04:17:16.026 INFO:tasks.rados.rados.0.smithi190.stdout:346: done (8 left) 2023-11-01T04:17:16.026 INFO:tasks.rados.rados.0.smithi190.stdout:347: done (7 left) 2023-11-01T04:17:16.027 INFO:tasks.rados.rados.0.smithi190.stdout:348: done (6 left) 2023-11-01T04:17:16.027 INFO:tasks.rados.rados.0.smithi190.stdout:351: done (5 left) 2023-11-01T04:17:16.027 INFO:tasks.rados.rados.0.smithi190.stdout:352: done (4 left) 2023-11-01T04:17:16.027 INFO:tasks.rados.rados.0.smithi190.stdout:355: done (3 left) 2023-11-01T04:17:16.028 INFO:tasks.rados.rados.0.smithi190.stdout:356: done (2 left) 2023-11-01T04:17:16.028 INFO:tasks.rados.rados.0.smithi190.stdout:357: rollback oid 21 current snap is 33 2023-11-01T04:17:16.028 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 21 to 29 2023-11-01T04:17:16.028 INFO:tasks.rados.rados.0.smithi190.stdout:358: rmattr oid 49 current snap is 33 2023-11-01T04:17:16.029 INFO:tasks.rados.rados.0.smithi190.stdout:358: done (3 left) 2023-11-01T04:17:16.029 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 39 current snap is 33 2023-11-01T04:17:16.029 INFO:tasks.rados.rados.0.smithi190.stdout:359: seq_num 131 ranges {2080768=655360,2736128=425984,3162112=638976,3801088=491520} 2023-11-01T04:17:16.030 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:17:16.026+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:17:16.049 INFO:tasks.rados.rados.0.smithi190.stdout:359: writing smithi19039307-39 from 2080768 to 2736128 tid 1 2023-11-01T04:17:16.053 INFO:tasks.rados.rados.0.smithi190.stdout:359: writing smithi19039307-39 from 2736128 to 3162112 tid 2 2023-11-01T04:17:16.058 INFO:tasks.rados.rados.0.smithi190.stdout:359: writing smithi19039307-39 from 3162112 to 3801088 tid 3 2023-11-01T04:17:16.063 INFO:tasks.rados.rados.0.smithi190.stdout:359: writing smithi19039307-39 from 3801088 to 4292608 tid 4 2023-11-01T04:17:16.063 INFO:tasks.rados.rados.0.smithi190.stdout:360: copy_from oid 49 from oid 36 current snap is 33 2023-11-01T04:17:16.063 INFO:tasks.rados.rados.0.smithi190.stdout:361: rmattr oid 45 current snap is 33 2023-11-01T04:17:16.063 INFO:tasks.rados.rados.0.smithi190.stdout:361: done (5 left) 2023-11-01T04:17:16.063 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 29 current snap is 33 2023-11-01T04:17:16.064 INFO:tasks.rados.rados.0.smithi190.stdout:362: seq_num 132 ranges {0=671744,671744=573440,1245184=753664,1998848=229376} 2023-11-01T04:17:16.068 INFO:tasks.rados.rados.0.smithi190.stdout:362: writing smithi19039307-29 from 0 to 671744 tid 1 2023-11-01T04:17:16.073 INFO:tasks.rados.rados.0.smithi190.stdout:362: writing smithi19039307-29 from 671744 to 1245184 tid 2 2023-11-01T04:17:16.082 INFO:tasks.rados.rados.0.smithi190.stdout:362: writing smithi19039307-29 from 1245184 to 1998848 tid 3 2023-11-01T04:17:16.082 INFO:tasks.rados.rados.0.smithi190.stdout:362: writing smithi19039307-29 from 1998848 to 2228224 tid 4 2023-11-01T04:17:16.083 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 24 current snap is 33 2023-11-01T04:17:16.083 INFO:tasks.rados.rados.0.smithi190.stdout:363: seq_num 133 ranges {3850240=688128,4538368=720896,5259264=475136,5734400=360448} 2023-11-01T04:17:16.111 INFO:tasks.rados.rados.0.smithi190.stdout:363: writing smithi19039307-24 from 3850240 to 4538368 tid 1 2023-11-01T04:17:16.115 INFO:tasks.rados.rados.0.smithi190.stdout:363: writing smithi19039307-24 from 4538368 to 5259264 tid 2 2023-11-01T04:17:16.118 INFO:tasks.rados.rados.0.smithi190.stdout:363: writing smithi19039307-24 from 5259264 to 5734400 tid 3 2023-11-01T04:17:16.120 INFO:tasks.rados.rados.0.smithi190.stdout:363: writing smithi19039307-24 from 5734400 to 6094848 tid 4 2023-11-01T04:17:16.121 INFO:tasks.rados.rados.0.smithi190.stdout:364: copy_from oid 41 from oid 38 current snap is 33 2023-11-01T04:17:16.121 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 32 current snap is 33 2023-11-01T04:17:16.121 INFO:tasks.rados.rados.0.smithi190.stdout:365: seq_num 134 ranges {114688=704512,819200=475136,1294336=573440,1867776=507904} 2023-11-01T04:17:16.125 INFO:tasks.rados.rados.0.smithi190.stdout:365: writing smithi19039307-32 from 114688 to 819200 tid 1 2023-11-01T04:17:16.128 INFO:tasks.rados.rados.0.smithi190.stdout:365: writing smithi19039307-32 from 819200 to 1294336 tid 2 2023-11-01T04:17:16.131 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:16.131 INFO:tasks.rados.rados.0.smithi190.stdout:365: writing smithi19039307-32 from 1294336 to 1867776 tid 3 2023-11-01T04:17:16.132 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:16.128+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:16.133 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:16.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-01T04:17:16.135 INFO:tasks.rados.rados.0.smithi190.stdout:365: writing smithi19039307-32 from 1867776 to 2375680 tid 4 2023-11-01T04:17:16.135 INFO:tasks.rados.rados.0.smithi190.stdout:366: snap_remove snap 30 2023-11-01T04:17:16.138 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:17:16.138 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-01T04:17:16.167 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:17:16.168 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-01T04:17:16.195 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:17:16.196 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-01T04:17:16.231 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:16.270 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:17:16.270 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-01T04:17:16.271 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:16.271 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:16.271 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.12 2023-11-01T04:17:16.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-01T04:17:16.298 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:17:16.298 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-01T04:17:16.325 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:17:16.326 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-01T04:17:16.334 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:17:16.332+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:17:16.352 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:17:16.353 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-01T04:17:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-01T04:17:16.359 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-01T04:17:16.412 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:16.413 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:16.427 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-01T04:17:16.472 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:17:16.472+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:17:16.536 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:17:16.536+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:17:16.548 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:16.548 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:16.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-01T04:17:16.637 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:17:16.634+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:17:16.728 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:16.729 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:16.737 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:17:16.732+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:17:16.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-01T04:17:16.838 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:17:16.835+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:17:16.867 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:16.867 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:16.881 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-01T04:17:16.939 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:17:16.935+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:17:17.003 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:17.003 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:17.017 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-11-01T04:17:17.040 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:17.038+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:17.043 INFO:tasks.rados.rados.0.smithi190.stdout:366: done (9 left) 2023-11-01T04:17:17.043 INFO:tasks.rados.rados.0.smithi190.stdout:367: copy_from oid 31 from oid 9 current snap is 33 2023-11-01T04:17:17.044 INFO:tasks.rados.rados.0.smithi190.stdout:354: expect (ObjNum 6 snap 0 seq_num 6) 2023-11-01T04:17:17.046 INFO:tasks.rados.rados.0.smithi190.stdout:365: finishing write tid 1 to smithi19039307-32 2023-11-01T04:17:17.047 INFO:tasks.rados.rados.0.smithi190.stdout:365: finishing write tid 2 to smithi19039307-32 2023-11-01T04:17:17.047 INFO:tasks.rados.rados.0.smithi190.stdout:360: finishing copy_from racing read to smithi19039307-49 2023-11-01T04:17:17.047 INFO:tasks.rados.rados.0.smithi190.stdout:362: finishing write tid 1 to smithi19039307-29 2023-11-01T04:17:17.047 INFO:tasks.rados.rados.0.smithi190.stdout:364: finishing copy_from racing read to smithi19039307-41 2023-11-01T04:17:17.047 INFO:tasks.rados.rados.0.smithi190.stdout:357: finishing rollback tid 0 to smithi19039307-21 2023-11-01T04:17:17.048 INFO:tasks.rados.rados.0.smithi190.stdout:362: finishing write tid 2 to smithi19039307-29 2023-11-01T04:17:17.048 INFO:tasks.rados.rados.0.smithi190.stdout:353: finishing write tid 6 to smithi19039307-22 2023-11-01T04:17:17.048 INFO:tasks.rados.rados.0.smithi190.stdout:353: oid 22 updating version 0 to 105 2023-11-01T04:17:17.048 INFO:tasks.rados.rados.0.smithi190.stdout:353: oid 22 version 105 is already newer than 104 2023-11-01T04:17:17.048 INFO:tasks.rados.rados.0.smithi190.stdout:353: oid 22 updating version 105 to 108 2023-11-01T04:17:17.048 INFO:tasks.rados.rados.0.smithi190.stdout:353: oid 22 version 108 is already newer than 106 2023-11-01T04:17:17.049 INFO:tasks.rados.rados.0.smithi190.stdout:353: oid 22 version 108 is already newer than 107 2023-11-01T04:17:17.049 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 22 v 108 (ObjNum 130 snap 32 seq_num 130) dirty exists 2023-11-01T04:17:17.049 INFO:tasks.rados.rados.0.smithi190.stdout:353: left oid 22 (ObjNum 130 snap 32 seq_num 130) 2023-11-01T04:17:17.049 INFO:tasks.rados.rados.0.smithi190.stdout:360: finishing copy_from to smithi19039307-49 2023-11-01T04:17:17.049 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 49 v 39 (ObjNum 53 snap 1 seq_num 53) dirty exists 2023-11-01T04:17:17.050 INFO:tasks.rados.rados.0.smithi190.stdout:365: finishing write tid 3 to smithi19039307-32 2023-11-01T04:17:17.050 INFO:tasks.rados.rados.0.smithi190.stdout:365: finishing write tid 4 to smithi19039307-32 2023-11-01T04:17:17.050 INFO:tasks.rados.rados.0.smithi190.stdout:353: done (9 left) 2023-11-01T04:17:17.050 INFO:tasks.rados.rados.0.smithi190.stdout:354: done (8 left) 2023-11-01T04:17:17.050 INFO:tasks.rados.rados.0.smithi190.stdout:360: done (7 left) 2023-11-01T04:17:17.051 INFO:tasks.rados.rados.0.smithi190.stdout:368: snap_remove snap 24 2023-11-01T04:17:17.141 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:17.138+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:17.156 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:17.156 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:17.170 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-11-01T04:17:17.242 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:17:17.239+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:17:17.294 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:17.294 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:17.308 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-11-01T04:17:17.343 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:17.340+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:17.444 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:17.442+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:17.477 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:17.477 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:17.491 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-11-01T04:17:17.544 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:17:17.542+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:17:17.613 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:17.613 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:17.627 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-11-01T04:17:17.645 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:17.640+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:17.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~23s 2023-11-01T04:17:17.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~15s 2023-11-01T04:17:17.745 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:17.748 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:17.748 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:17.762 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-11-01T04:17:17.847 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:17:17.844+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:17:17.884 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:17.884 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:17.899 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2023-11-01T04:17:17.947 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:17.944+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:18.024 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:18.024 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:18.037 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-11-01T04:17:18.039 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2023-11-01T04:17:18.048 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:18.044+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:18.057 INFO:tasks.rados.rados.0.smithi190.stdout:368: done (7 left) 2023-11-01T04:17:18.058 INFO:tasks.rados.rados.0.smithi190.stdout:369: snap_create 2023-11-01T04:17:18.058 INFO:tasks.rados.rados.0.smithi190.stdout:357: finishing rollback tid 1 to smithi19039307-21 2023-11-01T04:17:18.058 INFO:tasks.rados.rados.0.smithi190.stdout:365: finishing write tid 5 to smithi19039307-32 2023-11-01T04:17:18.058 INFO:tasks.rados.rados.0.smithi190.stdout:364: finishing copy_from to smithi19039307-41 2023-11-01T04:17:18.059 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 41 v 2 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-11-01T04:17:18.059 INFO:tasks.rados.rados.0.smithi190.stdout:357: finishing rollback tid 2 to smithi19039307-21 2023-11-01T04:17:18.059 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 21 v 35 (ObjNum 59 snap 5 seq_num 59) dirty exists 2023-11-01T04:17:18.059 INFO:tasks.rados.rados.0.smithi190.stdout:359: finishing write tid 1 to smithi19039307-39 2023-11-01T04:17:18.060 INFO:tasks.rados.rados.0.smithi190.stdout:359: finishing write tid 2 to smithi19039307-39 2023-11-01T04:17:18.060 INFO:tasks.rados.rados.0.smithi190.stdout:359: finishing write tid 3 to smithi19039307-39 2023-11-01T04:17:18.060 INFO:tasks.rados.rados.0.smithi190.stdout:362: finishing write tid 3 to smithi19039307-29 2023-11-01T04:17:18.060 INFO:tasks.rados.rados.0.smithi190.stdout:362: finishing write tid 4 to smithi19039307-29 2023-11-01T04:17:18.061 INFO:tasks.rados.rados.0.smithi190.stdout:362: finishing write tid 5 to smithi19039307-29 2023-11-01T04:17:18.061 INFO:tasks.rados.rados.0.smithi190.stdout:363: finishing write tid 1 to smithi19039307-24 2023-11-01T04:17:18.061 INFO:tasks.rados.rados.0.smithi190.stdout:363: finishing write tid 2 to smithi19039307-24 2023-11-01T04:17:18.061 INFO:tasks.rados.rados.0.smithi190.stdout:363: finishing write tid 3 to smithi19039307-24 2023-11-01T04:17:18.061 INFO:tasks.rados.rados.0.smithi190.stdout:363: finishing write tid 4 to smithi19039307-24 2023-11-01T04:17:18.062 INFO:tasks.rados.rados.0.smithi190.stdout:363: finishing write tid 5 to smithi19039307-24 2023-11-01T04:17:18.062 INFO:tasks.rados.rados.0.smithi190.stdout:359: finishing write tid 4 to smithi19039307-39 2023-11-01T04:17:18.062 INFO:tasks.rados.rados.0.smithi190.stdout:362: finishing write tid 6 to smithi19039307-29 2023-11-01T04:17:18.062 INFO:tasks.rados.rados.0.smithi190.stdout:362: oid 29 updating version 0 to 130 2023-11-01T04:17:18.063 INFO:tasks.rados.rados.0.smithi190.stdout:362: oid 29 version 130 is already newer than 129 2023-11-01T04:17:18.063 INFO:tasks.rados.rados.0.smithi190.stdout:362: oid 29 version 130 is already newer than 128 2023-11-01T04:17:18.063 INFO:tasks.rados.rados.0.smithi190.stdout:362: oid 29 updating version 130 to 131 2023-11-01T04:17:18.063 INFO:tasks.rados.rados.0.smithi190.stdout:362: oid 29 updating version 131 to 132 2023-11-01T04:17:18.063 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 29 v 132 (ObjNum 132 snap 33 seq_num 132) dirty exists 2023-11-01T04:17:18.064 INFO:tasks.rados.rados.0.smithi190.stdout:362: left oid 29 (ObjNum 132 snap 33 seq_num 132) 2023-11-01T04:17:18.064 INFO:tasks.rados.rados.0.smithi190.stdout:363: finishing write tid 6 to smithi19039307-24 2023-11-01T04:17:18.064 INFO:tasks.rados.rados.0.smithi190.stdout:363: oid 24 updating version 0 to 137 2023-11-01T04:17:18.064 INFO:tasks.rados.rados.0.smithi190.stdout:363: oid 24 version 137 is already newer than 136 2023-11-01T04:17:18.064 INFO:tasks.rados.rados.0.smithi190.stdout:363: oid 24 version 137 is already newer than 135 2023-11-01T04:17:18.065 INFO:tasks.rados.rados.0.smithi190.stdout:363: oid 24 version 137 is already newer than 134 2023-11-01T04:17:18.065 INFO:tasks.rados.rados.0.smithi190.stdout:363: oid 24 updating version 137 to 138 2023-11-01T04:17:18.065 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 24 v 138 (ObjNum 133 snap 33 seq_num 133) dirty exists 2023-11-01T04:17:18.065 INFO:tasks.rados.rados.0.smithi190.stdout:363: left oid 24 (ObjNum 133 snap 33 seq_num 133) 2023-11-01T04:17:18.065 INFO:tasks.rados.rados.0.smithi190.stdout:359: finishing write tid 5 to smithi19039307-39 2023-11-01T04:17:18.066 INFO:tasks.rados.rados.0.smithi190.stdout:365: finishing write tid 6 to smithi19039307-32 2023-11-01T04:17:18.066 INFO:tasks.rados.rados.0.smithi190.stdout:365: oid 32 updating version 0 to 103 2023-11-01T04:17:18.066 INFO:tasks.rados.rados.0.smithi190.stdout:365: oid 32 updating version 103 to 107 2023-11-01T04:17:18.066 INFO:tasks.rados.rados.0.smithi190.stdout:365: oid 32 version 107 is already newer than 106 2023-11-01T04:17:18.066 INFO:tasks.rados.rados.0.smithi190.stdout:365: oid 32 version 107 is already newer than 105 2023-11-01T04:17:18.067 INFO:tasks.rados.rados.0.smithi190.stdout:365: oid 32 version 107 is already newer than 104 2023-11-01T04:17:18.067 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 32 v 107 (ObjNum 134 snap 33 seq_num 134) dirty exists 2023-11-01T04:17:18.067 INFO:tasks.rados.rados.0.smithi190.stdout:365: left oid 32 (ObjNum 134 snap 33 seq_num 134) 2023-11-01T04:17:18.067 INFO:tasks.rados.rados.0.smithi190.stdout:359: finishing write tid 6 to smithi19039307-39 2023-11-01T04:17:18.067 INFO:tasks.rados.rados.0.smithi190.stdout:359: oid 39 updating version 0 to 73 2023-11-01T04:17:18.068 INFO:tasks.rados.rados.0.smithi190.stdout:359: oid 39 version 73 is already newer than 72 2023-11-01T04:17:18.068 INFO:tasks.rados.rados.0.smithi190.stdout:359: oid 39 updating version 73 to 74 2023-11-01T04:17:18.068 INFO:tasks.rados.rados.0.smithi190.stdout:359: oid 39 version 74 is already newer than 70 2023-11-01T04:17:18.068 INFO:tasks.rados.rados.0.smithi190.stdout:359: oid 39 version 74 is already newer than 71 2023-11-01T04:17:18.068 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 39 v 74 (ObjNum 131 snap 33 seq_num 131) dirty exists 2023-11-01T04:17:18.069 INFO:tasks.rados.rados.0.smithi190.stdout:359: left oid 39 (ObjNum 131 snap 33 seq_num 131) 2023-11-01T04:17:18.069 INFO:tasks.rados.rados.0.smithi190.stdout:367: finishing copy_from racing read to smithi19039307-31 2023-11-01T04:17:18.069 INFO:tasks.rados.rados.0.smithi190.stdout:367: finishing copy_from to smithi19039307-31 2023-11-01T04:17:18.069 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 31 v 17 (ObjNum 88 snap 20 seq_num 88) dirty exists 2023-11-01T04:17:18.148 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:18.196 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:18.196 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:18.210 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2023-11-01T04:17:18.249 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:18.292 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2023-11-01T04:17:18.326 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:18.327 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:18.341 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2023-11-01T04:17:18.352 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:18.350+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:18.360 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-01T04:17:18.404 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:18.406 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:18.407 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['config', 'set', 'filestore_inject_stall', '3'] 2023-11-01T04:17:18.452 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:17:18.450+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:17:18.484 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:18.486 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:18.487 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2023-11-01T04:17:18.523 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:17:18.524 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-11-01T04:17:18.544 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:17:18.544 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-01T04:17:18.553 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:18.548+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:18.564 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:17:18.564 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-01T04:17:18.584 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:17:18.584 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-01T04:17:18.588 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:18.591 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:18.592 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2023-11-01T04:17:18.600 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:17:18.600 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-01T04:17:18.620 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:17:18.620 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-01T04:17:18.640 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:17:18.641 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-01T04:17:18.641 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-11-01T04:17:18.654 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:18.650+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:18.657 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:17:18.657 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-01T04:17:18.680 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:17:18.680 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-01T04:17:18.702 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:18.705 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:18.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-11-01T04:17:18.755 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:17:18.752+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:17:18.838 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:18.838 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:17:18.838 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2023-11-01T04:17:18.839 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:18.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-11-01T04:17:18.894 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:17:18.892+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:17:18.955 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:18.950+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:18.978 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:17:18.979 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-01T04:17:19.011 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:17:19.011 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-01T04:17:19.011 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.12 2023-11-01T04:17:19.015 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:19.016 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:17:19.016 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:17:19.016 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:17:19.016 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:19.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-11-01T04:17:19.050 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:17:19.051 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-01T04:17:19.051 INFO:teuthology.orchestra.run.smithi153.stderr:noscrub is set 2023-11-01T04:17:19.057 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:17:19.056+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:17:19.060 INFO:tasks.rados.rados.0.smithi190.stdout:357: done (7 left) 2023-11-01T04:17:19.060 INFO:tasks.rados.rados.0.smithi190.stdout:359: done (6 left) 2023-11-01T04:17:19.060 INFO:tasks.rados.rados.0.smithi190.stdout:362: done (5 left) 2023-11-01T04:17:19.060 INFO:tasks.rados.rados.0.smithi190.stdout:363: done (4 left) 2023-11-01T04:17:19.061 INFO:tasks.rados.rados.0.smithi190.stdout:364: done (3 left) 2023-11-01T04:17:19.061 INFO:tasks.rados.rados.0.smithi190.stdout:365: done (2 left) 2023-11-01T04:17:19.061 INFO:tasks.rados.rados.0.smithi190.stdout:367: done (1 left) 2023-11-01T04:17:19.061 INFO:tasks.rados.rados.0.smithi190.stdout:369: done (0 left) 2023-11-01T04:17:19.062 INFO:tasks.rados.rados.0.smithi190.stdout:370: read oid 15 snap -1 2023-11-01T04:17:19.062 INFO:tasks.rados.rados.0.smithi190.stdout:370: expect (ObjNum 112 snap 25 seq_num 112) 2023-11-01T04:17:19.062 INFO:tasks.rados.rados.0.smithi190.stdout:371: copy_from oid 3 from oid 47 current snap is 34 2023-11-01T04:17:19.062 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 38 current snap is 34 2023-11-01T04:17:19.062 INFO:tasks.rados.rados.0.smithi190.stdout:372: seq_num 135 ranges {622592=720896,1343488=622592,1966080=688128,2654208=245760} 2023-11-01T04:17:19.076 INFO:tasks.rados.rados.0.smithi190.stdout:372: writing smithi19039307-38 from 622592 to 1343488 tid 1 2023-11-01T04:17:19.083 INFO:tasks.rados.rados.0.smithi190.stdout:372: writing smithi19039307-38 from 1343488 to 1966080 tid 2 2023-11-01T04:17:19.091 INFO:tasks.rados.rados.0.smithi190.stdout:372: writing smithi19039307-38 from 1966080 to 2654208 tid 3 2023-11-01T04:17:19.094 INFO:tasks.rados.rados.0.smithi190.stdout:372: writing smithi19039307-38 from 2654208 to 2899968 tid 4 2023-11-01T04:17:19.095 INFO:tasks.rados.rados.0.smithi190.stdout:373: read oid 39 snap -1 2023-11-01T04:17:19.095 INFO:tasks.rados.rados.0.smithi190.stdout:373: expect (ObjNum 131 snap 33 seq_num 131) 2023-11-01T04:17:19.095 INFO:tasks.rados.rados.0.smithi190.stdout:374: rmattr oid 46 current snap is 34 2023-11-01T04:17:19.095 INFO:tasks.rados.rados.0.smithi190.stdout:374: done (4 left) 2023-11-01T04:17:19.095 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 35 current snap is 34 2023-11-01T04:17:19.096 INFO:tasks.rados.rados.0.smithi190.stdout:375: seq_num 136 ranges {3080192=737280,3817472=770048,4587520=540672,5128192=245760} 2023-11-01T04:17:19.132 INFO:tasks.rados.rados.0.smithi190.stdout:375: writing smithi19039307-35 from 3080192 to 3817472 tid 1 2023-11-01T04:17:19.137 INFO:tasks.rados.rados.0.smithi190.stdout:375: writing smithi19039307-35 from 3817472 to 4587520 tid 2 2023-11-01T04:17:19.141 INFO:tasks.rados.rados.0.smithi190.stdout:375: writing smithi19039307-35 from 4587520 to 5128192 tid 3 2023-11-01T04:17:19.143 INFO:tasks.rados.rados.0.smithi190.stdout:375: writing smithi19039307-35 from 5128192 to 5373952 tid 4 2023-11-01T04:17:19.143 INFO:tasks.rados.rados.0.smithi190.stdout:376: read oid 10 snap -1 2023-11-01T04:17:19.144 INFO:tasks.rados.rados.0.smithi190.stdout:376: expect (ObjNum 66 snap 6 seq_num 66) 2023-11-01T04:17:19.144 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 5 current snap is 34 2023-11-01T04:17:19.144 INFO:tasks.rados.rados.0.smithi190.stdout:377: seq_num 137 ranges {1818624=753664,2572288=786432,3358720=655360,4014080=114688} 2023-11-01T04:17:19.157 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:19.154+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:19.159 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:19.159 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:17:19.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:17:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:17:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.4s3 37/31 ECSubWrite(tid=22, reqid=client.4559.0:197, at_version=37'9, trim_to=0'0, roll_forward_to=35'6))", 2023-11-01T04:17:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:17:19.160 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 62.116047930000001, 2023-11-01T04:17:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.301677274, 2023-11-01T04:17:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:17:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040272+0000", 2023-11-01T04:17:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040273+0000", 2023-11-01T04:17:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3370000000000001e-06 2023-11-01T04:17:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040307+0000", 2023-11-01T04:17:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3714999999999998e-05 2023-11-01T04:17:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.040331+0000", 2023-11-01T04:17:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4409e-05 2023-11-01T04:17:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.140479+0000", 2023-11-01T04:17:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10014764900000001 2023-11-01T04:17:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.240734+0000", 2023-11-01T04:17:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100255645 2023-11-01T04:17:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.240751+0000", 2023-11-01T04:17:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6852000000000001e-05 2023-11-01T04:17:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341894+0000", 2023-11-01T04:17:19.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101143302 2023-11-01T04:17:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.341949+0000", 2023-11-01T04:17:19.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4364999999999999e-05 2023-11-01T04:17:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:647 3.9s1 3:9c07f031:::smithi19039307-43:head [create,setxattr ]PaibOQQZPXlPQ_TVRVsq`]`lQnrU_adXdhR`ia_ (33) in=40b,setxattr mga[\\pjjlr_f^pjpoNm[SiQinb`p]Uljsr\\PZamg (1614) in=40b,setxattr \\\\nNdqV]P_aoSQ]T][SZi`\\`emqRalsThkWkp]_O (1998) in=40b,setxattr QPRVcO^PYksRnjf^RqtasbqiemoSYpeimc]_kQOo (1505) in=40b,setxattr i`dWfsdYVZUqQmWleQeniob`plYiVWUWSsnsd`\\a (1017) in=40b,setxattr Qn`WN^VrYNfSm_nWkOlO[bZ_[Zci__kgsaZeg[_R (1614) in=40b,setxattr \\]NdaY]dOTpk\\bgtmgifQtgWtaqfd]r`TbktPRjh (19016) in=40b,setxattr NlQZaoabhi (349) in=10b,setxattr QbZ^bqgnd]QnP\\ZjeRjaP[UfZgdm[V^sRaRpnZYY (1963) in=40b,setxattr `]YWO_jgNfUs_SQ`RaqUhXmNY[TtZZbaUj\\NPtY^ (1190) in=40b,setxattr dnfqsNhbe]^ml]ekS[ST`fnk_kpiTUa]Y]ges`rW (17824) in=40b,setxattr RgOUnWhYeYcgnikgiQQWNr_OWg_k`YSaN^XQOdPc (1437) in=40b,setxattr QUmNqZgRZOU]WaWWgl\\pnRhtilqS`cQaScmgUp_e (1438) in=40b,setxattr TfW^Smr_geRfti]efnPdr`iWQUikeXOp`StYqtQd (12280) in=40b,setxattr ZtZ (349) in=3b,setxattr NQYfjd\\ (349) in=7b,setxattr aNaTblXeObdSN[ahZn`j[mcbUaSjeXOnc[dZ^ZZo (910) in=40b,setxattr deb]iS_OUlbmWR_sP^er[TdaV`cSqnnthSotq]YZ (7928) in=40b,setxattr ^s_gVdo]kXUSXb]aqYjbUYbRmcoYksXSWQdVsh (349) in=38b,setxattr h^YTbW\\dli\\^dffQaXRNaZQVkZPONqhhabmhR_[s (1822) in=40b] snapc 28=[28,27,26,22,1b,19,18,12] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e78)", 2023-11-01T04:17:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:58.841255+0000", 2023-11-01T04:17:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 20.315064577000001, 2023-11-01T04:17:19.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.24042176, 2023-11-01T04:17:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:19.173 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 647 2023-11-01T04:17:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.841255+0000", 2023-11-01T04:17:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.841255+0000", 2023-11-01T04:17:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.841258+0000", 2023-11-01T04:17:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7099999999999999e-06 2023-11-01T04:17:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.841499+0000", 2023-11-01T04:17:19.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000241017 2023-11-01T04:17:19.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.841527+0000", 2023-11-01T04:17:19.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8622000000000001e-05 2023-11-01T04:17:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.841555+0000", 2023-11-01T04:17:19.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7798e-05 2023-11-01T04:17:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.841732+0000", 2023-11-01T04:17:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017735900000000001 2023-11-01T04:17:19.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op must wait for map", 2023-11-01T04:17:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.841758+0000", 2023-11-01T04:17:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5165999999999999e-05 2023-11-01T04:17:19.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.948792+0000", 2023-11-01T04:17:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.107034474 2023-11-01T04:17:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2023-11-01T04:17:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:58.948798+0000", 2023-11-01T04:17:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3860000000000004e-06 2023-11-01T04:17:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:00.081195+0000", 2023-11-01T04:17:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.132396741 2023-11-01T04:17:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:00.081677+0000", 2023-11-01T04:17:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00048148700000000001 2023-11-01T04:17:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.2s5 81/78 ECSubWrite(tid=367, reqid=client.4559.0:679, at_version=81'111, trim_to=0'0, roll_forward_to=0'0))", 2023-11-01T04:17:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:02.460637+0000", 2023-11-01T04:17:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 16.695682454, 2023-11-01T04:17:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30519870300000002, 2023-11-01T04:17:19.186 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.460637+0000", 2023-11-01T04:17:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.460637+0000", 2023-11-01T04:17:19.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.460639+0000", 2023-11-01T04:17:19.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5829999999999999e-06 2023-11-01T04:17:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.460874+0000", 2023-11-01T04:17:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000235178 2023-11-01T04:17:19.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.460907+0000", 2023-11-01T04:17:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2957999999999999e-05 2023-11-01T04:17:19.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.561048+0000", 2023-11-01T04:17:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100141626 2023-11-01T04:17:19.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.561180+0000", 2023-11-01T04:17:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013203100000000001 2023-11-01T04:17:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.561189+0000", 2023-11-01T04:17:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1640000000000001e-06 2023-11-01T04:17:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.765764+0000", 2023-11-01T04:17:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.204575857 2023-11-01T04:17:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.765836+0000", 2023-11-01T04:17:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1305999999999998e-05 2023-11-01T04:17:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.196 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.2s5 81/78 ECSubWrite(tid=371, reqid=client.4559.0:687, at_version=81'116, trim_to=0'0, roll_forward_to=0'0))", 2023-11-01T04:17:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:02.561958+0000", 2023-11-01T04:17:19.197 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 16.594361498000001, 2023-11-01T04:17:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20862663000000001, 2023-11-01T04:17:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.561958+0000", 2023-11-01T04:17:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.561958+0000", 2023-11-01T04:17:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.561959+0000", 2023-11-01T04:17:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5300000000000003e-07 2023-11-01T04:17:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.562255+0000", 2023-11-01T04:17:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029631500000000001 2023-11-01T04:17:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.562279+0000", 2023-11-01T04:17:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3459e-05 2023-11-01T04:17:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.562297+0000", 2023-11-01T04:17:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.825e-05 2023-11-01T04:17:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.764954+0000", 2023-11-01T04:17:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20265675999999999 2023-11-01T04:17:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.764968+0000", 2023-11-01T04:17:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4819e-05 2023-11-01T04:17:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.770538+0000", 2023-11-01T04:17:19.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0055699529999999999 2023-11-01T04:17:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:02.770585+0000", 2023-11-01T04:17:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6320999999999999e-05 2023-11-01T04:17:19.207 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.208 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:696 3.9s1 3:959ff18f:::smithi19039307-32:head [copy-from ver 14] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:03.411212+0000", 2023-11-01T04:17:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 15.745107965999999, 2023-11-01T04:17:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72798311000000004, 2023-11-01T04:17:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.209 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 696 2023-11-01T04:17:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.210 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.411212+0000", 2023-11-01T04:17:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.411212+0000", 2023-11-01T04:17:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.411215+0000", 2023-11-01T04:17:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.416e-06 2023-11-01T04:17:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.411231+0000", 2023-11-01T04:17:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5693e-05 2023-11-01T04:17:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.411263+0000", 2023-11-01T04:17:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2075e-05 2023-11-01T04:17:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.411277+0000", 2023-11-01T04:17:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.456e-05 2023-11-01T04:17:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.411465+0000", 2023-11-01T04:17:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018794300000000001 2023-11-01T04:17:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.411862+0000", 2023-11-01T04:17:19.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00039678499999999998 2023-11-01T04:17:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.733731+0000", 2023-11-01T04:17:19.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32186899499999999 2023-11-01T04:17:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.937186+0000", 2023-11-01T04:17:19.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20345480199999999 2023-11-01T04:17:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:04.138851+0000", 2023-11-01T04:17:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20166492699999999 2023-11-01T04:17:19.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:04.139195+0000", 2023-11-01T04:17:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034391400000000001 2023-11-01T04:17:19.221 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:697 3.9s1 3:959ff18f:::smithi19039307-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:19.222 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:03.411319+0000", 2023-11-01T04:17:19.223 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 15.74500096, 2023-11-01T04:17:19.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42523839899999999, 2023-11-01T04:17:19.223 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.223 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:19.223 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:19.224 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:19.224 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:19.224 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 697 2023-11-01T04:17:19.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.224 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.411319+0000", 2023-11-01T04:17:19.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.411319+0000", 2023-11-01T04:17:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.411320+0000", 2023-11-01T04:17:19.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3999999999999999e-06 2023-11-01T04:17:19.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.411326+0000", 2023-11-01T04:17:19.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7830000000000004e-06 2023-11-01T04:17:19.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.411364+0000", 2023-11-01T04:17:19.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8368000000000003e-05 2023-11-01T04:17:19.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.411389+0000", 2023-11-01T04:17:19.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5313e-05 2023-11-01T04:17:19.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.512283+0000", 2023-11-01T04:17:19.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100893994 2023-11-01T04:17:19.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for blocked object", 2023-11-01T04:17:19.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.512344+0000", 2023-11-01T04:17:19.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0922999999999999e-05 2023-11-01T04:17:19.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.835877+0000", 2023-11-01T04:17:19.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32353243999999998 2023-11-01T04:17:19.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:19.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.836066+0000", 2023-11-01T04:17:19.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018959000000000001 2023-11-01T04:17:19.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.836557+0000", 2023-11-01T04:17:19.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00049058800000000003 2023-11-01T04:17:19.234 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.234 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.235 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.235 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:711 3.9s1 3:94ead3c8:::smithi19039307-33:head [append 0~0] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:19.235 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:03.453875+0000", 2023-11-01T04:17:19.235 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 15.702444695000001, 2023-11-01T04:17:19.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48559091199999999, 2023-11-01T04:17:19.236 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.236 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:19.236 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:19.236 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:19.237 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:19.237 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 711 2023-11-01T04:17:19.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.237 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.453875+0000", 2023-11-01T04:17:19.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.453875+0000", 2023-11-01T04:17:19.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.453878+0000", 2023-11-01T04:17:19.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2399999999999999e-06 2023-11-01T04:17:19.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.453893+0000", 2023-11-01T04:17:19.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4766000000000001e-05 2023-11-01T04:17:19.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.453931+0000", 2023-11-01T04:17:19.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8377999999999997e-05 2023-11-01T04:17:19.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.453951+0000", 2023-11-01T04:17:19.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9623000000000002e-05 2023-11-01T04:17:19.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.512374+0000", 2023-11-01T04:17:19.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.058423244999999999 2023-11-01T04:17:19.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:19.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.512501+0000", 2023-11-01T04:17:19.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012673799999999999 2023-11-01T04:17:19.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.513216+0000", 2023-11-01T04:17:19.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00071495900000000004 2023-11-01T04:17:19.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.515845+0000", 2023-11-01T04:17:19.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026294090000000001 2023-11-01T04:17:19.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:19.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.939107+0000", 2023-11-01T04:17:19.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42326225099999998 2023-11-01T04:17:19.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.939466+0000", 2023-11-01T04:17:19.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00035830299999999997 2023-11-01T04:17:19.248 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.248 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.248 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.249 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:712 3.9s1 3:94ead3c8:::smithi19039307-33:head [rollback 19] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:19.249 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:03.453993+0000", 2023-11-01T04:17:19.249 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 15.702326676, 2023-11-01T04:17:19.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.485984054, 2023-11-01T04:17:19.249 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.249 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:19.250 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:19.250 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:19.250 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:19.250 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 712 2023-11-01T04:17:19.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.250 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.453993+0000", 2023-11-01T04:17:19.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.453993+0000", 2023-11-01T04:17:19.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.453994+0000", 2023-11-01T04:17:19.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.19e-06 2023-11-01T04:17:19.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.453999+0000", 2023-11-01T04:17:19.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9699999999999998e-06 2023-11-01T04:17:19.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.454017+0000", 2023-11-01T04:17:19.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8219e-05 2023-11-01T04:17:19.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.454035+0000", 2023-11-01T04:17:19.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7929000000000001e-05 2023-11-01T04:17:19.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.513934+0000", 2023-11-01T04:17:19.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.059898711 2023-11-01T04:17:19.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:19.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.514027+0000", 2023-11-01T04:17:19.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2746999999999998e-05 2023-11-01T04:17:19.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.514668+0000", 2023-11-01T04:17:19.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00064185800000000003 2023-11-01T04:17:19.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.515848+0000", 2023-11-01T04:17:19.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001179191 2023-11-01T04:17:19.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:19.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.939661+0000", 2023-11-01T04:17:19.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42381337400000002 2023-11-01T04:17:19.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.939977+0000", 2023-11-01T04:17:19.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00031586500000000003 2023-11-01T04:17:19.261 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.261 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.261 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.262 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:713 3.9s1 3:94ead3c8:::smithi19039307-33:head [append 0~0] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:19.262 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:03.454057+0000", 2023-11-01T04:17:19.262 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 15.702262315, 2023-11-01T04:17:19.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.584067267, 2023-11-01T04:17:19.262 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.262 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:19.263 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:19.263 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:19.263 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:19.263 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 713 2023-11-01T04:17:19.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.263 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.454057+0000", 2023-11-01T04:17:19.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.454057+0000", 2023-11-01T04:17:19.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.454058+0000", 2023-11-01T04:17:19.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2330000000000001e-06 2023-11-01T04:17:19.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.454063+0000", 2023-11-01T04:17:19.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6199999999999998e-06 2023-11-01T04:17:19.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.454074+0000", 2023-11-01T04:17:19.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0526000000000001e-05 2023-11-01T04:17:19.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.454081+0000", 2023-11-01T04:17:19.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7800000000000001e-06 2023-11-01T04:17:19.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.515215+0000", 2023-11-01T04:17:19.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.061133727999999998 2023-11-01T04:17:19.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:19.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.515281+0000", 2023-11-01T04:17:19.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5785999999999997e-05 2023-11-01T04:17:19.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.515554+0000", 2023-11-01T04:17:19.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00027337900000000003 2023-11-01T04:17:19.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.516261+0000", 2023-11-01T04:17:19.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00070689400000000001 2023-11-01T04:17:19.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:19.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:04.037478+0000", 2023-11-01T04:17:19.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52121641200000002 2023-11-01T04:17:19.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:04.038124+0000", 2023-11-01T04:17:19.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00064690899999999998 2023-11-01T04:17:19.274 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.275 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.275 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.275 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.9s1 82/78 ECSubWriteReply(tid=2, last_complete=82'90, committed=1, applied=1))", 2023-11-01T04:17:19.275 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:03.630474+0000", 2023-11-01T04:17:19.276 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 15.525845616, 2023-11-01T04:17:19.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30632458699999998, 2023-11-01T04:17:19.276 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.276 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:19.276 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.630474+0000", 2023-11-01T04:17:19.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.630474+0000", 2023-11-01T04:17:19.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.630474+0000", 2023-11-01T04:17:19.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1e-07 2023-11-01T04:17:19.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.630476+0000", 2023-11-01T04:17:19.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.125e-06 2023-11-01T04:17:19.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.630484+0000", 2023-11-01T04:17:19.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5839999999999997e-06 2023-11-01T04:17:19.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.630491+0000", 2023-11-01T04:17:19.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8240000000000002e-06 2023-11-01T04:17:19.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.936724+0000", 2023-11-01T04:17:19.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30623283000000001 2023-11-01T04:17:19.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.936798+0000", 2023-11-01T04:17:19.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4913999999999995e-05 2023-11-01T04:17:19.283 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.283 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.284 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.284 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.1s3 82/78 ECSubWrite(tid=112, reqid=client.4559.0:735, at_version=82'66, trim_to=0'0, roll_forward_to=0'0))", 2023-11-01T04:17:19.284 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:03.641106+0000", 2023-11-01T04:17:19.284 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 15.515212988, 2023-11-01T04:17:19.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.296062035, 2023-11-01T04:17:19.285 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.285 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:19.285 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.641106+0000", 2023-11-01T04:17:19.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.641106+0000", 2023-11-01T04:17:19.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.641107+0000", 2023-11-01T04:17:19.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1999999999999999e-07 2023-11-01T04:17:19.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.641220+0000", 2023-11-01T04:17:19.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000113232 2023-11-01T04:17:19.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.641235+0000", 2023-11-01T04:17:19.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4234e-05 2023-11-01T04:17:19.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.641242+0000", 2023-11-01T04:17:19.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5299999999999999e-06 2023-11-01T04:17:19.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.732334+0000", 2023-11-01T04:17:19.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.091091462999999998 2023-11-01T04:17:19.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.732362+0000", 2023-11-01T04:17:19.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8748999999999999e-05 2023-11-01T04:17:19.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.937114+0000", 2023-11-01T04:17:19.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.204751722 2023-11-01T04:17:19.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.937169+0000", 2023-11-01T04:17:19.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4385000000000002e-05 2023-11-01T04:17:19.294 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.294 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.294 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.295 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.9s1 82/78 ECSubWriteReply(tid=0, last_complete=82'88, committed=1, applied=1))", 2023-11-01T04:17:19.295 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:03.716509+0000", 2023-11-01T04:17:19.295 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 15.439810786000001, 2023-11-01T04:17:19.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22044992399999999, 2023-11-01T04:17:19.295 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.295 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:19.296 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.716509+0000", 2023-11-01T04:17:19.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.716509+0000", 2023-11-01T04:17:19.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.716510+0000", 2023-11-01T04:17:19.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.767e-06 2023-11-01T04:17:19.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.716520+0000", 2023-11-01T04:17:19.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8090000000000004e-06 2023-11-01T04:17:19.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.716547+0000", 2023-11-01T04:17:19.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7098999999999999e-05 2023-11-01T04:17:19.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.716562+0000", 2023-11-01T04:17:19.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.505e-05 2023-11-01T04:17:19.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.936891+0000", 2023-11-01T04:17:19.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.220329046 2023-11-01T04:17:19.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.936959+0000", 2023-11-01T04:17:19.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7152999999999999e-05 2023-11-01T04:17:19.302 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.303 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.303 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.303 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.9s1 82/78 ECSubWriteReply(tid=1, last_complete=82'89, committed=1, applied=1))", 2023-11-01T04:17:19.303 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:03.716590+0000", 2023-11-01T04:17:19.304 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 15.439729291999999, 2023-11-01T04:17:19.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22059968499999999, 2023-11-01T04:17:19.304 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.304 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:19.304 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.716590+0000", 2023-11-01T04:17:19.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.716590+0000", 2023-11-01T04:17:19.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.716591+0000", 2023-11-01T04:17:19.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8700000000000005e-07 2023-11-01T04:17:19.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.716594+0000", 2023-11-01T04:17:19.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5530000000000001e-06 2023-11-01T04:17:19.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.716605+0000", 2023-11-01T04:17:19.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0742e-05 2023-11-01T04:17:19.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.716641+0000", 2023-11-01T04:17:19.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6216e-05 2023-11-01T04:17:19.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.937068+0000", 2023-11-01T04:17:19.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.220426238 2023-11-01T04:17:19.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.937190+0000", 2023-11-01T04:17:19.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122249 2023-11-01T04:17:19.312 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.312 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.312 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.312 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.9s1 82/78 ECSubWriteReply(tid=2, last_complete=82'90, committed=1, applied=1))", 2023-11-01T04:17:19.313 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:03.717721+0000", 2023-11-01T04:17:19.313 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 15.438598925000001, 2023-11-01T04:17:19.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21955291099999999, 2023-11-01T04:17:19.313 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.313 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:19.314 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.717721+0000", 2023-11-01T04:17:19.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.717721+0000", 2023-11-01T04:17:19.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.717722+0000", 2023-11-01T04:17:19.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.643e-06 2023-11-01T04:17:19.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.717731+0000", 2023-11-01T04:17:19.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.867e-06 2023-11-01T04:17:19.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.717760+0000", 2023-11-01T04:17:19.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8529000000000001e-05 2023-11-01T04:17:19.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.717777+0000", 2023-11-01T04:17:19.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7635999999999999e-05 2023-11-01T04:17:19.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.937242+0000", 2023-11-01T04:17:19.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.219464398 2023-11-01T04:17:19.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.937273+0000", 2023-11-01T04:17:19.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1838000000000002e-05 2023-11-01T04:17:19.321 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.321 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.322 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.322 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.2s5 89/78 ECSubWrite(tid=384, reqid=client.4559.0:781, at_version=89'120, trim_to=0'0, roll_forward_to=86'118))", 2023-11-01T04:17:19.322 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:10.164413+0000", 2023-11-01T04:17:19.322 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 8.9919068929999995, 2023-11-01T04:17:19.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30941946199999998, 2023-11-01T04:17:19.323 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.323 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:19.323 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.164413+0000", 2023-11-01T04:17:19.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.164413+0000", 2023-11-01T04:17:19.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.164414+0000", 2023-11-01T04:17:19.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1799999999999999e-06 2023-11-01T04:17:19.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.164807+0000", 2023-11-01T04:17:19.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00039327399999999999 2023-11-01T04:17:19.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.164869+0000", 2023-11-01T04:17:19.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2410999999999996e-05 2023-11-01T04:17:19.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265018+0000", 2023-11-01T04:17:19.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10014882999999999 2023-11-01T04:17:19.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.365317+0000", 2023-11-01T04:17:19.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10029823 2023-11-01T04:17:19.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.365351+0000", 2023-11-01T04:17:19.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4462000000000003e-05 2023-11-01T04:17:19.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.473774+0000", 2023-11-01T04:17:19.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.108423017 2023-11-01T04:17:19.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.473832+0000", 2023-11-01T04:17:19.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8057999999999999e-05 2023-11-01T04:17:19.332 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.332 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.332 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.333 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.2s5 89/78 ECSubWrite(tid=385, reqid=client.4559.0:782, at_version=89'121, trim_to=0'0, roll_forward_to=86'118))", 2023-11-01T04:17:19.333 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:10.265071+0000", 2023-11-01T04:17:19.333 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 8.8912487999999996, 2023-11-01T04:17:19.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20880597100000001, 2023-11-01T04:17:19.334 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.334 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:19.334 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265071+0000", 2023-11-01T04:17:19.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265071+0000", 2023-11-01T04:17:19.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265072+0000", 2023-11-01T04:17:19.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.023e-06 2023-11-01T04:17:19.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265331+0000", 2023-11-01T04:17:19.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025961199999999999 2023-11-01T04:17:19.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265356+0000", 2023-11-01T04:17:19.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4572000000000001e-05 2023-11-01T04:17:19.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265369+0000", 2023-11-01T04:17:19.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.306e-05 2023-11-01T04:17:19.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.369094+0000", 2023-11-01T04:17:19.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10372492799999999 2023-11-01T04:17:19.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.369122+0000", 2023-11-01T04:17:19.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8442999999999999e-05 2023-11-01T04:17:19.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.473841+0000", 2023-11-01T04:17:19.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10471902700000001 2023-11-01T04:17:19.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.473877+0000", 2023-11-01T04:17:19.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5305999999999997e-05 2023-11-01T04:17:19.343 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.343 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.343 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.344 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.2s5 89/78 ECSubWrite(tid=387, reqid=client.4559.0:784, at_version=89'123, trim_to=0'0, roll_forward_to=86'118))", 2023-11-01T04:17:19.344 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:10.265548+0000", 2023-11-01T04:17:19.344 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 8.8907715849999995, 2023-11-01T04:17:19.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.209002102, 2023-11-01T04:17:19.344 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.344 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:19.345 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265548+0000", 2023-11-01T04:17:19.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265548+0000", 2023-11-01T04:17:19.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265548+0000", 2023-11-01T04:17:19.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7299999999999996e-07 2023-11-01T04:17:19.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265603+0000", 2023-11-01T04:17:19.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4548e-05 2023-11-01T04:17:19.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265619+0000", 2023-11-01T04:17:19.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6180000000000001e-05 2023-11-01T04:17:19.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265629+0000", 2023-11-01T04:17:19.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4360000000000008e-06 2023-11-01T04:17:19.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.472209+0000", 2023-11-01T04:17:19.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.206579975 2023-11-01T04:17:19.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.472239+0000", 2023-11-01T04:17:19.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0358999999999999e-05 2023-11-01T04:17:19.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.474478+0000", 2023-11-01T04:17:19.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002238636 2023-11-01T04:17:19.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.474550+0000", 2023-11-01T04:17:19.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2395000000000001e-05 2023-11-01T04:17:19.354 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.354 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.354 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.355 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.2s5 89/78 ECSubWrite(tid=388, reqid=client.4559.0:785, at_version=89'124, trim_to=0'0, roll_forward_to=86'118))", 2023-11-01T04:17:19.355 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:10.265648+0000", 2023-11-01T04:17:19.355 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 8.8906714119999997, 2023-11-01T04:17:19.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20976177100000001, 2023-11-01T04:17:19.355 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.355 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:19.356 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265648+0000", 2023-11-01T04:17:19.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265648+0000", 2023-11-01T04:17:19.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265649+0000", 2023-11-01T04:17:19.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6400000000000002e-07 2023-11-01T04:17:19.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265668+0000", 2023-11-01T04:17:19.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9799000000000001e-05 2023-11-01T04:17:19.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.265683+0000", 2023-11-01T04:17:19.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4403000000000001e-05 2023-11-01T04:17:19.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.365814+0000", 2023-11-01T04:17:19.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100131605 2023-11-01T04:17:19.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.473341+0000", 2023-11-01T04:17:19.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.107526792 2023-11-01T04:17:19.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.473361+0000", 2023-11-01T04:17:19.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0112999999999999e-05 2023-11-01T04:17:19.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.475341+0000", 2023-11-01T04:17:19.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019793509999999999 2023-11-01T04:17:19.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.475410+0000", 2023-11-01T04:17:19.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9144000000000001e-05 2023-11-01T04:17:19.366 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.366 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.366 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.367 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:847 3.9s1 3:94ead3c8:::smithi19039307-33:head [copy-from ver 17] snapc 39=[39,38,37,35,34,2f,28,19] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2023-11-01T04:17:19.367 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:15.018461+0000", 2023-11-01T04:17:19.367 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 4.1378583899999999, 2023-11-01T04:17:19.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.421746966, 2023-11-01T04:17:19.367 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.368 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:19.368 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:19.368 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:19.368 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:19.368 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 847 2023-11-01T04:17:19.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.369 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.018461+0000", 2023-11-01T04:17:19.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.018461+0000", 2023-11-01T04:17:19.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.018464+0000", 2023-11-01T04:17:19.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0130000000000002e-06 2023-11-01T04:17:19.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.018507+0000", 2023-11-01T04:17:19.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3018e-05 2023-11-01T04:17:19.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.018561+0000", 2023-11-01T04:17:19.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4058999999999999e-05 2023-11-01T04:17:19.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.018578+0000", 2023-11-01T04:17:19.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7306e-05 2023-11-01T04:17:19.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.018830+0000", 2023-11-01T04:17:19.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025139299999999998 2023-11-01T04:17:19.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:19.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.019183+0000", 2023-11-01T04:17:19.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00035328699999999998 2023-11-01T04:17:19.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.376 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.234880+0000", 2023-11-01T04:17:19.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.215696837 2023-11-01T04:17:19.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.239265+0000", 2023-11-01T04:17:19.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043854310000000004 2023-11-01T04:17:19.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:19.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.439848+0000", 2023-11-01T04:17:19.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.200582125 2023-11-01T04:17:19.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.440208+0000", 2023-11-01T04:17:19.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036049699999999999 2023-11-01T04:17:19.379 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.379 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.379 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.380 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:848 3.9s1 3:94ead3c8:::smithi19039307-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2023-11-01T04:17:19.380 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:15.018618+0000", 2023-11-01T04:17:19.380 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 4.1377011619999999, 2023-11-01T04:17:19.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22027002300000001, 2023-11-01T04:17:19.381 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.381 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:19.381 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:19.381 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:19.381 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:19.382 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 848 2023-11-01T04:17:19.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.382 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.018618+0000", 2023-11-01T04:17:19.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.018618+0000", 2023-11-01T04:17:19.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.018619+0000", 2023-11-01T04:17:19.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.17e-06 2023-11-01T04:17:19.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.018626+0000", 2023-11-01T04:17:19.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0499999999999997e-06 2023-11-01T04:17:19.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.018641+0000", 2023-11-01T04:17:19.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5610000000000001e-05 2023-11-01T04:17:19.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.387 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.387 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.387 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.018651+0000", 2023-11-01T04:17:19.387 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0023e-05 2023-11-01T04:17:19.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.388 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.388 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.118810+0000", 2023-11-01T04:17:19.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100158603 2023-11-01T04:17:19.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.389 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for blocked object", 2023-11-01T04:17:19.389 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.118919+0000", 2023-11-01T04:17:19.389 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000109083 2023-11-01T04:17:19.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.389 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.390 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.390 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.238178+0000", 2023-11-01T04:17:19.390 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.119259511 2023-11-01T04:17:19.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.390 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.391 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:19.391 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.238431+0000", 2023-11-01T04:17:19.391 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025239300000000001 2023-11-01T04:17:19.391 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.391 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.391 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.392 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:15.238888+0000", 2023-11-01T04:17:19.392 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00045758000000000002 2023-11-01T04:17:19.392 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.392 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.392 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.392 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.393 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.393 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:19.394 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:17:19.395 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-01T04:17:19.395 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:17:19.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-11-01T04:17:19.397 INFO:tasks.rados.rados.0.smithi190.stdout:377: writing smithi19039307-5 from 1818624 to 2572288 tid 1 2023-11-01T04:17:19.397 INFO:tasks.rados.rados.0.smithi190.stdout:377: writing smithi19039307-5 from 2572288 to 3358720 tid 2 2023-11-01T04:17:19.397 INFO:tasks.rados.rados.0.smithi190.stdout:377: writing smithi19039307-5 from 3358720 to 4014080 tid 3 2023-11-01T04:17:19.398 INFO:tasks.rados.rados.0.smithi190.stdout:377: writing smithi19039307-5 from 4014080 to 4128768 tid 4 2023-11-01T04:17:19.398 INFO:tasks.rados.rados.0.smithi190.stdout:378: rmattr oid 12 current snap is 34 2023-11-01T04:17:19.398 INFO:tasks.rados.rados.0.smithi190.stdout:379: read oid 20 snap -1 2023-11-01T04:17:19.398 INFO:tasks.rados.rados.0.smithi190.stdout:379: expect (ObjNum 110 snap 24 seq_num 110) 2023-11-01T04:17:19.398 INFO:tasks.rados.rados.0.smithi190.stdout:380: snap_create 2023-11-01T04:17:19.398 INFO:tasks.rados.rados.0.smithi190.stdout:372: finishing write tid 1 to smithi19039307-38 2023-11-01T04:17:19.399 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:17:19.392+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:17:19.399 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-01T04:17:19.400 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-01T04:17:19.400 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-01T04:17:19.454 INFO:tasks.rados.rados.0.smithi190.stdout:376: expect (ObjNum 66 snap 6 seq_num 66) 2023-11-01T04:17:19.494 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:19.492+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:19.532 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:19.532 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:17:19.533 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2023-11-01T04:17:19.533 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:19.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-11-01T04:17:19.564 INFO:tasks.rados.rados.0.smithi190.stdout:372: finishing write tid 2 to smithi19039307-38 2023-11-01T04:17:19.565 INFO:tasks.rados.rados.0.smithi190.stdout:377: finishing write tid 1 to smithi19039307-5 2023-11-01T04:17:19.565 INFO:tasks.rados.rados.0.smithi190.stdout:377: finishing write tid 2 to smithi19039307-5 2023-11-01T04:17:19.565 INFO:tasks.rados.rados.0.smithi190.stdout:377: finishing write tid 3 to smithi19039307-5 2023-11-01T04:17:19.565 INFO:tasks.rados.rados.0.smithi190.stdout:372: finishing write tid 3 to smithi19039307-38 2023-11-01T04:17:19.565 INFO:tasks.rados.rados.0.smithi190.stdout:377: finishing write tid 4 to smithi19039307-5 2023-11-01T04:17:19.566 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 12 v 83 (ObjNum 11 snap 0 seq_num 11) dirty exists 2023-11-01T04:17:19.574 INFO:tasks.rados.rados.0.smithi190.stdout:377: finishing write tid 5 to smithi19039307-5 2023-11-01T04:17:19.587 INFO:tasks.rados.rados.0.smithi190.stdout:372: finishing write tid 4 to smithi19039307-38 2023-11-01T04:17:19.588 INFO:tasks.rados.rados.0.smithi190.stdout:372: finishing write tid 5 to smithi19039307-38 2023-11-01T04:17:19.634 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:17:19.632+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:17:19.675 INFO:tasks.rados.rados.0.smithi190.stdout:370: expect (ObjNum 112 snap 25 seq_num 112) 2023-11-01T04:17:19.696 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:17:19.696+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:17:19.714 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:19.714 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:17:19.715 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:17:19.715 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:17:19.715 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:19.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-11-01T04:17:19.780 INFO:tasks.rados.rados.0.smithi190.stdout:371: finishing copy_from racing read to smithi19039307-3 2023-11-01T04:17:19.780 INFO:tasks.rados.rados.0.smithi190.stdout:375: finishing write tid 1 to smithi19039307-35 2023-11-01T04:17:19.780 INFO:tasks.rados.rados.0.smithi190.stdout:375: finishing write tid 2 to smithi19039307-35 2023-11-01T04:17:19.781 INFO:tasks.rados.rados.0.smithi190.stdout:375: finishing write tid 3 to smithi19039307-35 2023-11-01T04:17:19.781 INFO:tasks.rados.rados.0.smithi190.stdout:375: finishing write tid 4 to smithi19039307-35 2023-11-01T04:17:19.781 INFO:tasks.rados.rados.0.smithi190.stdout:377: finishing write tid 6 to smithi19039307-5 2023-11-01T04:17:19.781 INFO:tasks.rados.rados.0.smithi190.stdout:377: oid 5 updating version 0 to 78 2023-11-01T04:17:19.781 INFO:tasks.rados.rados.0.smithi190.stdout:377: oid 5 updating version 78 to 82 2023-11-01T04:17:19.782 INFO:tasks.rados.rados.0.smithi190.stdout:377: oid 5 version 82 is already newer than 79 2023-11-01T04:17:19.782 INFO:tasks.rados.rados.0.smithi190.stdout:377: oid 5 version 82 is already newer than 80 2023-11-01T04:17:19.782 INFO:tasks.rados.rados.0.smithi190.stdout:377: oid 5 version 82 is already newer than 81 2023-11-01T04:17:19.782 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 5 v 82 (ObjNum 137 snap 34 seq_num 137) dirty exists 2023-11-01T04:17:19.782 INFO:tasks.rados.rados.0.smithi190.stdout:377: left oid 5 (ObjNum 137 snap 34 seq_num 137) 2023-11-01T04:17:19.814 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:17:19.812+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:17:19.863 INFO:tasks.rados.rados.0.smithi190.stdout:375: finishing write tid 5 to smithi19039307-35 2023-11-01T04:17:19.882 INFO:tasks.rados.rados.0.smithi190.stdout:372: finishing write tid 6 to smithi19039307-38 2023-11-01T04:17:19.883 INFO:tasks.rados.rados.0.smithi190.stdout:372: oid 38 updating version 0 to 24 2023-11-01T04:17:19.883 INFO:tasks.rados.rados.0.smithi190.stdout:372: oid 38 updating version 24 to 26 2023-11-01T04:17:19.883 INFO:tasks.rados.rados.0.smithi190.stdout:372: oid 38 updating version 26 to 27 2023-11-01T04:17:19.883 INFO:tasks.rados.rados.0.smithi190.stdout:372: oid 38 updating version 27 to 28 2023-11-01T04:17:19.883 INFO:tasks.rados.rados.0.smithi190.stdout:372: oid 38 version 28 is already newer than 25 2023-11-01T04:17:19.884 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 38 v 28 (ObjNum 135 snap 34 seq_num 135) dirty exists 2023-11-01T04:17:19.884 INFO:tasks.rados.rados.0.smithi190.stdout:372: left oid 38 (ObjNum 135 snap 34 seq_num 135) 2023-11-01T04:17:19.886 INFO:tasks.rados.rados.0.smithi190.stdout:371: finishing copy_from to smithi19039307-3 2023-11-01T04:17:19.887 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 3 v 36 (ObjNum 120 snap 31 seq_num 120) dirty exists 2023-11-01T04:17:19.893 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:19.893 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:17:19.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:17:19.893 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:17:19.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.894 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:166 3.8s0 3:1b945fb3:::smithi19039307-3:head [read 0~49152,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2023-11-01T04:17:19.894 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.317340+0000", 2023-11-01T04:17:19.894 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 63.574947952999999, 2023-11-01T04:17:19.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.405887145, 2023-11-01T04:17:19.895 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.895 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:19.895 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:19.895 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:19.895 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:19.896 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 166 2023-11-01T04:17:19.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.896 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317340+0000", 2023-11-01T04:17:19.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317340+0000", 2023-11-01T04:17:19.898 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.898 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.898 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317341+0000", 2023-11-01T04:17:19.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.077e-06 2023-11-01T04:17:19.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.899 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.899 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.899 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317350+0000", 2023-11-01T04:17:19.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0329999999999997e-06 2023-11-01T04:17:19.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317367+0000", 2023-11-01T04:17:19.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6612999999999998e-05 2023-11-01T04:17:19.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.317380+0000", 2023-11-01T04:17:19.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2832e-05 2023-11-01T04:17:19.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.421004+0000", 2023-11-01T04:17:19.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.103624181 2023-11-01T04:17:19.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:19.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.421215+0000", 2023-11-01T04:17:19.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00021057199999999999 2023-11-01T04:17:19.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.723227+0000", 2023-11-01T04:17:19.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30201283699999998 2023-11-01T04:17:19.905 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.905 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.905 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.905 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:326 3.8s0 3:1514c4b8:::smithi19039307-15:head [append 0~376832] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:17:19.906 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.056621+0000", 2023-11-01T04:17:19.906 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.835667049999998, 2023-11-01T04:17:19.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43382757900000002, 2023-11-01T04:17:19.906 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.906 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:19.907 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:19.907 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:19.907 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:19.907 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 326 2023-11-01T04:17:19.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.908 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.056621+0000", 2023-11-01T04:17:19.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.056621+0000", 2023-11-01T04:17:19.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.056623+0000", 2023-11-01T04:17:19.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2640000000000001e-06 2023-11-01T04:17:19.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057077+0000", 2023-11-01T04:17:19.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00045417599999999998 2023-11-01T04:17:19.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057092+0000", 2023-11-01T04:17:19.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5138e-05 2023-11-01T04:17:19.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057103+0000", 2023-11-01T04:17:19.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1117999999999999e-05 2023-11-01T04:17:19.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.159853+0000", 2023-11-01T04:17:19.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10274968399999999 2023-11-01T04:17:19.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:19.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.159948+0000", 2023-11-01T04:17:19.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5097000000000001e-05 2023-11-01T04:17:19.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.160626+0000", 2023-11-01T04:17:19.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00067830199999999998 2023-11-01T04:17:19.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162313+0000", 2023-11-01T04:17:19.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016867799999999999 2023-11-01T04:17:19.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:19.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.490341+0000", 2023-11-01T04:17:19.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32802831100000002 2023-11-01T04:17:19.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.490449+0000", 2023-11-01T04:17:19.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010770899999999999 2023-11-01T04:17:19.919 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.919 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.919 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.920 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:327 3.8s0 3:1514c4b8:::smithi19039307-15:head [setxattr _header (56) in=63b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:17:19.920 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.057121+0000", 2023-11-01T04:17:19.920 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.835166989999998, 2023-11-01T04:17:19.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.63439970300000004, 2023-11-01T04:17:19.920 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.921 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:19.921 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:19.921 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:19.921 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:19.921 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 327 2023-11-01T04:17:19.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.922 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057121+0000", 2023-11-01T04:17:19.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057121+0000", 2023-11-01T04:17:19.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057122+0000", 2023-11-01T04:17:19.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8199999999999999e-07 2023-11-01T04:17:19.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057126+0000", 2023-11-01T04:17:19.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0439999999999997e-06 2023-11-01T04:17:19.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057134+0000", 2023-11-01T04:17:19.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.456e-06 2023-11-01T04:17:19.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057141+0000", 2023-11-01T04:17:19.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.861e-06 2023-11-01T04:17:19.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.161271+0000", 2023-11-01T04:17:19.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104129201 2023-11-01T04:17:19.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:19.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.161389+0000", 2023-11-01T04:17:19.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011806 2023-11-01T04:17:19.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.161906+0000", 2023-11-01T04:17:19.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00051756700000000003 2023-11-01T04:17:19.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162616+0000", 2023-11-01T04:17:19.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00070943799999999995 2023-11-01T04:17:19.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:19.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691275+0000", 2023-11-01T04:17:19.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52865958499999999 2023-11-01T04:17:19.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691521+0000", 2023-11-01T04:17:19.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024580899999999999 2023-11-01T04:17:19.933 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.933 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.934 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.934 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:328 3.8s0 3:1514c4b8:::smithi19039307-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:17:19.934 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.057152+0000", 2023-11-01T04:17:19.935 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.835136443000003, 2023-11-01T04:17:19.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83879504599999999, 2023-11-01T04:17:19.935 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.935 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:19.935 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:19.936 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:19.936 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:19.936 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 328 2023-11-01T04:17:19.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.936 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057152+0000", 2023-11-01T04:17:19.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057152+0000", 2023-11-01T04:17:19.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057152+0000", 2023-11-01T04:17:19.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5599999999999995e-07 2023-11-01T04:17:19.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057155+0000", 2023-11-01T04:17:19.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5050000000000002e-06 2023-11-01T04:17:19.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057163+0000", 2023-11-01T04:17:19.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1629999999999996e-06 2023-11-01T04:17:19.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.057169+0000", 2023-11-01T04:17:19.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8930000000000002e-06 2023-11-01T04:17:19.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162181+0000", 2023-11-01T04:17:19.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105011939 2023-11-01T04:17:19.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:19.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.162296+0000", 2023-11-01T04:17:19.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115392 2023-11-01T04:17:19.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691675+0000", 2023-11-01T04:17:19.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52937891999999998 2023-11-01T04:17:19.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:19.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.691837+0000", 2023-11-01T04:17:19.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016180099999999999 2023-11-01T04:17:19.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.895947+0000", 2023-11-01T04:17:19.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.204109877 2023-11-01T04:17:19.947 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.947 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.947 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.948 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:339 3.8s0 3:19d11dee:::smithi19039307-39:head [append 0~737280 in=737280b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:17:19.948 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.185406+0000", 2023-11-01T04:17:19.948 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.706882516, 2023-11-01T04:17:19.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70798112199999996, 2023-11-01T04:17:19.949 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.949 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:19.949 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:19.949 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:19.949 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:19.949 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 339 2023-11-01T04:17:19.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.950 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.185406+0000", 2023-11-01T04:17:19.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.185406+0000", 2023-11-01T04:17:19.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.185408+0000", 2023-11-01T04:17:19.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9530000000000002e-06 2023-11-01T04:17:19.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186621+0000", 2023-11-01T04:17:19.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012132270000000001 2023-11-01T04:17:19.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186644+0000", 2023-11-01T04:17:19.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2878e-05 2023-11-01T04:17:19.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186663+0000", 2023-11-01T04:17:19.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9650999999999999e-05 2023-11-01T04:17:19.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186754+0000", 2023-11-01T04:17:19.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0606999999999995e-05 2023-11-01T04:17:19.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:19.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186902+0000", 2023-11-01T04:17:19.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000147448 2023-11-01T04:17:19.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.188122+0000", 2023-11-01T04:17:19.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001220732 2023-11-01T04:17:19.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.191051+0000", 2023-11-01T04:17:19.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029283669999999999 2023-11-01T04:17:19.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:19.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893331+0000", 2023-11-01T04:17:19.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70228026200000004 2023-11-01T04:17:19.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893387+0000", 2023-11-01T04:17:19.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5996999999999999e-05 2023-11-01T04:17:19.961 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.961 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.961 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.962 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:340 3.8s0 3:19d11dee:::smithi19039307-39:head [append 0~376832 in=376832b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:17:19.962 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.186694+0000", 2023-11-01T04:17:19.962 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.705594667, 2023-11-01T04:17:19.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70680890500000004, 2023-11-01T04:17:19.963 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.963 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:19.963 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:19.963 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:19.963 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:19.963 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 340 2023-11-01T04:17:19.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.964 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186694+0000", 2023-11-01T04:17:19.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186694+0000", 2023-11-01T04:17:19.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.186695+0000", 2023-11-01T04:17:19.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3400000000000001e-06 2023-11-01T04:17:19.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187348+0000", 2023-11-01T04:17:19.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00065316300000000001 2023-11-01T04:17:19.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187364+0000", 2023-11-01T04:17:19.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5979999999999999e-05 2023-11-01T04:17:19.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187372+0000", 2023-11-01T04:17:19.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6599999999999995e-06 2023-11-01T04:17:19.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.189420+0000", 2023-11-01T04:17:19.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020482959999999998 2023-11-01T04:17:19.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:19.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.189500+0000", 2023-11-01T04:17:19.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9621000000000002e-05 2023-11-01T04:17:19.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.190386+0000", 2023-11-01T04:17:19.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00088641100000000003 2023-11-01T04:17:19.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.191553+0000", 2023-11-01T04:17:19.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001166413 2023-11-01T04:17:19.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:19.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893466+0000", 2023-11-01T04:17:19.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.701913911 2023-11-01T04:17:19.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893503+0000", 2023-11-01T04:17:19.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6109999999999998e-05 2023-11-01T04:17:19.975 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.975 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.975 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.976 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:341 3.8s0 3:19d11dee:::smithi19039307-39:head [setxattr _header (56) in=63b] snapc b=[b,4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:17:19.976 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.187391+0000", 2023-11-01T04:17:19.976 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.704897076999998, 2023-11-01T04:17:19.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70629933600000006, 2023-11-01T04:17:19.977 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.977 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:19.977 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:19.977 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:19.978 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:19.978 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 341 2023-11-01T04:17:19.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.978 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187391+0000", 2023-11-01T04:17:19.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187391+0000", 2023-11-01T04:17:19.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187392+0000", 2023-11-01T04:17:19.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7100000000000011e-07 2023-11-01T04:17:19.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187398+0000", 2023-11-01T04:17:19.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3329999999999996e-06 2023-11-01T04:17:19.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187407+0000", 2023-11-01T04:17:19.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6269999999999995e-06 2023-11-01T04:17:19.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187419+0000", 2023-11-01T04:17:19.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2336e-05 2023-11-01T04:17:19.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.265764+0000", 2023-11-01T04:17:19.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.078345021000000001 2023-11-01T04:17:19.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:19.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.265926+0000", 2023-11-01T04:17:19.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016117900000000001 2023-11-01T04:17:19.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:19.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.266578+0000", 2023-11-01T04:17:19.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00065216799999999995 2023-11-01T04:17:19.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:19.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.267885+0000", 2023-11-01T04:17:19.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013072699999999999 2023-11-01T04:17:19.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:19.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893617+0000", 2023-11-01T04:17:19.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62573200299999998 2023-11-01T04:17:19.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:19.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893691+0000", 2023-11-01T04:17:19.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3428000000000002e-05 2023-11-01T04:17:19.989 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.990 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:19.990 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:19.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.990 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:342 3.8s0 3:19d11dee:::smithi19039307-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2023-11-01T04:17:19.991 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.187433+0000", 2023-11-01T04:17:19.991 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.704855150999997, 2023-11-01T04:17:19.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.009816936, 2023-11-01T04:17:19.991 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:19.991 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:19.992 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:19.992 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:19.992 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:19.992 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 342 2023-11-01T04:17:19.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.993 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:19.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:19.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187433+0000", 2023-11-01T04:17:19.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187433+0000", 2023-11-01T04:17:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:19.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187434+0000", 2023-11-01T04:17:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.75e-07 2023-11-01T04:17:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187437+0000", 2023-11-01T04:17:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1870000000000001e-06 2023-11-01T04:17:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.187446+0000", 2023-11-01T04:17:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0499999999999997e-06 2023-11-01T04:17:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:19.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.287537+0000", 2023-11-01T04:17:19.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10009085 2023-11-01T04:17:19.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.288584+0000", 2023-11-01T04:17:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001047463 2023-11-01T04:17:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:19.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.288883+0000", 2023-11-01T04:17:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029857700000000001 2023-11-01T04:17:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.894227+0000", 2023-11-01T04:17:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60534433899999995 2023-11-01T04:17:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:20.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.894404+0000", 2023-11-01T04:17:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000176732 2023-11-01T04:17:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.197250+0000", 2023-11-01T04:17:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.302846163 2023-11-01T04:17:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=28, last_complete=46'20, committed=1, applied=1))", 2023-11-01T04:17:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.390586+0000", 2023-11-01T04:17:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.501702264999999, 2023-11-01T04:17:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30212400700000003, 2023-11-01T04:17:20.004 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390586+0000", 2023-11-01T04:17:20.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390586+0000", 2023-11-01T04:17:20.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390587+0000", 2023-11-01T04:17:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9999999999999997e-07 2023-11-01T04:17:20.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390592+0000", 2023-11-01T04:17:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9400000000000001e-06 2023-11-01T04:17:20.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390607+0000", 2023-11-01T04:17:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5216e-05 2023-11-01T04:17:20.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.390617+0000", 2023-11-01T04:17:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8260000000000005e-06 2023-11-01T04:17:20.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.692681+0000", 2023-11-01T04:17:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30206392300000001 2023-11-01T04:17:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.692710+0000", 2023-11-01T04:17:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9502e-05 2023-11-01T04:17:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.012 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=29, last_complete=46'21, committed=1, applied=1))", 2023-11-01T04:17:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.391131+0000", 2023-11-01T04:17:20.013 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.501157128999999, 2023-11-01T04:17:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30166817800000001, 2023-11-01T04:17:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391131+0000", 2023-11-01T04:17:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391131+0000", 2023-11-01T04:17:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391132+0000", 2023-11-01T04:17:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5000000000000001e-07 2023-11-01T04:17:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391138+0000", 2023-11-01T04:17:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1999999999999999e-06 2023-11-01T04:17:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391156+0000", 2023-11-01T04:17:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7252999999999999e-05 2023-11-01T04:17:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.391165+0000", 2023-11-01T04:17:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3530000000000001e-06 2023-11-01T04:17:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.692767+0000", 2023-11-01T04:17:20.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30160192600000002 2023-11-01T04:17:20.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.692799+0000", 2023-11-01T04:17:20.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2496000000000002e-05 2023-11-01T04:17:20.022 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.022 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.022 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:17:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:17:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.499949721, 2023-11-01T04:17:20.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40054957200000002, 2023-11-01T04:17:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:17:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:17:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392339+0000", 2023-11-01T04:17:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7400000000000002e-07 2023-11-01T04:17:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392345+0000", 2023-11-01T04:17:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8690000000000001e-06 2023-11-01T04:17:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392362+0000", 2023-11-01T04:17:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6194000000000001e-05 2023-11-01T04:17:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.392372+0000", 2023-11-01T04:17:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0692e-05 2023-11-01T04:17:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792857+0000", 2023-11-01T04:17:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40048482800000001 2023-11-01T04:17:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792888+0000", 2023-11-01T04:17:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1115000000000003e-05 2023-11-01T04:17:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=28, last_complete=46'20, committed=1, applied=1))", 2023-11-01T04:17:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.393704+0000", 2023-11-01T04:17:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.498584203999997, 2023-11-01T04:17:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39925663099999997, 2023-11-01T04:17:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393704+0000", 2023-11-01T04:17:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393704+0000", 2023-11-01T04:17:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393705+0000", 2023-11-01T04:17:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3699999999999999e-07 2023-11-01T04:17:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393711+0000", 2023-11-01T04:17:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6899999999999997e-06 2023-11-01T04:17:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393726+0000", 2023-11-01T04:17:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5559000000000001e-05 2023-11-01T04:17:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393735+0000", 2023-11-01T04:17:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2830000000000001e-06 2023-11-01T04:17:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792931+0000", 2023-11-01T04:17:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39919643599999999 2023-11-01T04:17:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792961+0000", 2023-11-01T04:17:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9726e-05 2023-11-01T04:17:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=29, last_complete=46'21, committed=1, applied=1))", 2023-11-01T04:17:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:17:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.498525325999999, 2023-11-01T04:17:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39925449800000001, 2023-11-01T04:17:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:17:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:17:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393763+0000", 2023-11-01T04:17:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7700000000000005e-07 2023-11-01T04:17:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393770+0000", 2023-11-01T04:17:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.28e-06 2023-11-01T04:17:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393779+0000", 2023-11-01T04:17:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6760000000000002e-06 2023-11-01T04:17:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.393786+0000", 2023-11-01T04:17:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8229999999999997e-06 2023-11-01T04:17:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.792996+0000", 2023-11-01T04:17:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39921008299999999 2023-11-01T04:17:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.793017+0000", 2023-11-01T04:17:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1158999999999999e-05 2023-11-01T04:17:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:17:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.394729+0000", 2023-11-01T04:17:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.497559420000002, 2023-11-01T04:17:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.49843099299999999, 2023-11-01T04:17:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394729+0000", 2023-11-01T04:17:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394729+0000", 2023-11-01T04:17:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394730+0000", 2023-11-01T04:17:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.093e-06 2023-11-01T04:17:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394738+0000", 2023-11-01T04:17:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2300000000000008e-06 2023-11-01T04:17:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.394760+0000", 2023-11-01T04:17:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1679999999999999e-05 2023-11-01T04:17:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.395298+0000", 2023-11-01T04:17:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053837199999999996 2023-11-01T04:17:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893130+0000", 2023-11-01T04:17:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.49783206600000002 2023-11-01T04:17:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893160+0000", 2023-11-01T04:17:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9552000000000001e-05 2023-11-01T04:17:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:17:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.468843+0000", 2023-11-01T04:17:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.423445569999998, 2023-11-01T04:17:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42441382100000002, 2023-11-01T04:17:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468843+0000", 2023-11-01T04:17:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468843+0000", 2023-11-01T04:17:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468845+0000", 2023-11-01T04:17:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.21e-06 2023-11-01T04:17:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468857+0000", 2023-11-01T04:17:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1678999999999999e-05 2023-11-01T04:17:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.468888+0000", 2023-11-01T04:17:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1788999999999997e-05 2023-11-01T04:17:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569025+0000", 2023-11-01T04:17:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100136424 2023-11-01T04:17:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893225+0000", 2023-11-01T04:17:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32420036899999999 2023-11-01T04:17:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893257+0000", 2023-11-01T04:17:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1350000000000003e-05 2023-11-01T04:17:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=28, last_complete=46'20, committed=1, applied=1))", 2023-11-01T04:17:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.569161+0000", 2023-11-01T04:17:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.323127292000002, 2023-11-01T04:17:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32424099299999998, 2023-11-01T04:17:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569161+0000", 2023-11-01T04:17:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569161+0000", 2023-11-01T04:17:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569162+0000", 2023-11-01T04:17:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3400000000000001e-06 2023-11-01T04:17:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569173+0000", 2023-11-01T04:17:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0293e-05 2023-11-01T04:17:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569197+0000", 2023-11-01T04:17:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4156e-05 2023-11-01T04:17:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569213+0000", 2023-11-01T04:17:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6455000000000001e-05 2023-11-01T04:17:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893271+0000", 2023-11-01T04:17:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32405778099999999 2023-11-01T04:17:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893402+0000", 2023-11-01T04:17:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000130968 2023-11-01T04:17:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=29, last_complete=46'21, committed=1, applied=1))", 2023-11-01T04:17:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.569263+0000", 2023-11-01T04:17:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.323024812, 2023-11-01T04:17:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32424595, 2023-11-01T04:17:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569263+0000", 2023-11-01T04:17:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569263+0000", 2023-11-01T04:17:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569264+0000", 2023-11-01T04:17:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7700000000000004e-07 2023-11-01T04:17:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569269+0000", 2023-11-01T04:17:20.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9490000000000004e-06 2023-11-01T04:17:20.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569282+0000", 2023-11-01T04:17:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3169999999999999e-05 2023-11-01T04:17:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569293+0000", 2023-11-01T04:17:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0526000000000001e-05 2023-11-01T04:17:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893418+0000", 2023-11-01T04:17:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32412480999999999 2023-11-01T04:17:20.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893509+0000", 2023-11-01T04:17:20.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1817999999999994e-05 2023-11-01T04:17:20.087 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 46/31 ECSubWriteReply(tid=30, last_complete=46'22, committed=1, applied=1))", 2023-11-01T04:17:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:26.569435+0000", 2023-11-01T04:17:20.089 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.322852838000003, 2023-11-01T04:17:20.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32468964500000003, 2023-11-01T04:17:20.089 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.089 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.089 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569435+0000", 2023-11-01T04:17:20.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569435+0000", 2023-11-01T04:17:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569436+0000", 2023-11-01T04:17:20.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6700000000000003e-07 2023-11-01T04:17:20.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569441+0000", 2023-11-01T04:17:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3059999999999997e-06 2023-11-01T04:17:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.569461+0000", 2023-11-01T04:17:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9856000000000001e-05 2023-11-01T04:17:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.669600+0000", 2023-11-01T04:17:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100139114 2023-11-01T04:17:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.893551+0000", 2023-11-01T04:17:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.223950963 2023-11-01T04:17:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:26.894125+0000", 2023-11-01T04:17:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00057373899999999998 2023-11-01T04:17:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(osd.12.22:5 3.8s0 3:19d11dee:::smithi19039307-39:head [assert-version v22,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2023-11-01T04:17:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.454207+0000", 2023-11-01T04:17:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 52.438080888999998, 2023-11-01T04:17:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35587170800000001, 2023-11-01T04:17:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454207+0000", 2023-11-01T04:17:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454207+0000", 2023-11-01T04:17:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454210+0000", 2023-11-01T04:17:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2299999999999998e-06 2023-11-01T04:17:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454219+0000", 2023-11-01T04:17:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0469999999999999e-06 2023-11-01T04:17:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454239+0000", 2023-11-01T04:17:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0499000000000001e-05 2023-11-01T04:17:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454256+0000", 2023-11-01T04:17:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6993000000000001e-05 2023-11-01T04:17:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454343+0000", 2023-11-01T04:17:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.687e-05 2023-11-01T04:17:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.454435+0000", 2023-11-01T04:17:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2019000000000001e-05 2023-11-01T04:17:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.810079+0000", 2023-11-01T04:17:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35564404999999999 2023-11-01T04:17:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.0s4 68/31 ECSubWrite(tid=36, reqid=osd.3.0:36, at_version=68'21, trim_to=0'0, roll_forward_to=67'19))", 2023-11-01T04:17:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.894669+0000", 2023-11-01T04:17:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 31.997618808999999, 2023-11-01T04:17:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31005245300000001, 2023-11-01T04:17:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894669+0000", 2023-11-01T04:17:20.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894669+0000", 2023-11-01T04:17:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894670+0000", 2023-11-01T04:17:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9399999999999995e-07 2023-11-01T04:17:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894722+0000", 2023-11-01T04:17:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1962000000000002e-05 2023-11-01T04:17:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894772+0000", 2023-11-01T04:17:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9771999999999997e-05 2023-11-01T04:17:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.994910+0000", 2023-11-01T04:17:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100138189 2023-11-01T04:17:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.104098+0000", 2023-11-01T04:17:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10918811 2023-11-01T04:17:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.104110+0000", 2023-11-01T04:17:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1787000000000001e-05 2023-11-01T04:17:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.204645+0000", 2023-11-01T04:17:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100534978 2023-11-01T04:17:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.204722+0000", 2023-11-01T04:17:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7161000000000002e-05 2023-11-01T04:17:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.118 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:20.119 INFO:tasks.rados.rados.0.smithi190.stdout:379: expect (ObjNum 110 snap 24 seq_num 110) 2023-11-01T04:17:20.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2023-11-01T04:17:20.121 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:17:20.118+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:17:20.135 INFO:tasks.rados.rados.0.smithi190.stdout:370: done (9 left) 2023-11-01T04:17:20.135 INFO:tasks.rados.rados.0.smithi190.stdout:371: done (8 left) 2023-11-01T04:17:20.135 INFO:tasks.rados.rados.0.smithi190.stdout:372: done (7 left) 2023-11-01T04:17:20.135 INFO:tasks.rados.rados.0.smithi190.stdout:376: done (6 left) 2023-11-01T04:17:20.136 INFO:tasks.rados.rados.0.smithi190.stdout:377: done (5 left) 2023-11-01T04:17:20.136 INFO:tasks.rados.rados.0.smithi190.stdout:378: done (4 left) 2023-11-01T04:17:20.136 INFO:tasks.rados.rados.0.smithi190.stdout:379: done (3 left) 2023-11-01T04:17:20.136 INFO:tasks.rados.rados.0.smithi190.stdout:380: done (2 left) 2023-11-01T04:17:20.136 INFO:tasks.rados.rados.0.smithi190.stdout:381: delete oid 32 current snap is 35 2023-11-01T04:17:20.168 INFO:tasks.rados.rados.0.smithi190.stdout:375: finishing write tid 6 to smithi19039307-35 2023-11-01T04:17:20.168 INFO:tasks.rados.rados.0.smithi190.stdout:375: oid 35 updating version 0 to 112 2023-11-01T04:17:20.168 INFO:tasks.rados.rados.0.smithi190.stdout:375: oid 35 updating version 112 to 114 2023-11-01T04:17:20.168 INFO:tasks.rados.rados.0.smithi190.stdout:375: oid 35 version 114 is already newer than 110 2023-11-01T04:17:20.169 INFO:tasks.rados.rados.0.smithi190.stdout:375: oid 35 version 114 is already newer than 111 2023-11-01T04:17:20.169 INFO:tasks.rados.rados.0.smithi190.stdout:375: oid 35 version 114 is already newer than 113 2023-11-01T04:17:20.169 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 35 v 114 (ObjNum 136 snap 34 seq_num 136) dirty exists 2023-11-01T04:17:20.169 INFO:tasks.rados.rados.0.smithi190.stdout:375: left oid 35 (ObjNum 136 snap 34 seq_num 136) 2023-11-01T04:17:20.202 INFO:tasks.rados.rados.0.smithi190.stdout:373: expect (ObjNum 131 snap 33 seq_num 131) 2023-11-01T04:17:20.220 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:20.218+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:20.294 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:20.294 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:17:20.294 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2023-11-01T04:17:20.294 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:20.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2023-11-01T04:17:20.346 INFO:tasks.rados.rados.0.smithi190.stdout:373: done (2 left) 2023-11-01T04:17:20.346 INFO:tasks.rados.rados.0.smithi190.stdout:375: done (1 left) 2023-11-01T04:17:20.346 INFO:tasks.rados.rados.0.smithi190.stdout:381: done (0 left) 2023-11-01T04:17:20.346 INFO:tasks.rados.rados.0.smithi190.stdout:382: rollback oid 18 current snap is 35 2023-11-01T04:17:20.347 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 18 to 33 2023-11-01T04:17:20.347 INFO:tasks.rados.rados.0.smithi190.stdout:383: snap_create 2023-11-01T04:17:20.358 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:20.356+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:20.423 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:17:20.423+0000 7f767a1e2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:17:20.429 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:20.429 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:17:20.430 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:17:20.430 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:17:20.430 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:20.445 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2023-11-01T04:17:20.523 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:17:20.520+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:17:20.553 INFO:tasks.rados.rados.0.smithi190.stdout:382: finishing rollback tid 0 to smithi19039307-18 2023-11-01T04:17:20.553 INFO:tasks.rados.rados.0.smithi190.stdout:382: finishing rollback tid 1 to smithi19039307-18 2023-11-01T04:17:20.554 INFO:tasks.rados.rados.0.smithi190.stdout:382: finishing rollback tid 2 to smithi19039307-18 2023-11-01T04:17:20.554 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 18 v 142 (ObjNum 98 snap 21 seq_num 98) dirty exists 2023-11-01T04:17:20.610 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:20.610 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:17:20.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:17:20.611 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:17:20.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.611 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.6s6 36/31 ECSubWrite(tid=22, reqid=client.4559.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=36'4))", 2023-11-01T04:17:20.612 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:15.688060+0000", 2023-11-01T04:17:20.612 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 64.920104973999997, 2023-11-01T04:17:20.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19558595100000001, 2023-11-01T04:17:20.612 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.612 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.612 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688060+0000", 2023-11-01T04:17:20.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688060+0000", 2023-11-01T04:17:20.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688062+0000", 2023-11-01T04:17:20.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1859999999999999e-06 2023-11-01T04:17:20.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688084+0000", 2023-11-01T04:17:20.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1849e-05 2023-11-01T04:17:20.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688115+0000", 2023-11-01T04:17:20.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0788999999999999e-05 2023-11-01T04:17:20.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.688131+0000", 2023-11-01T04:17:20.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6506000000000001e-05 2023-11-01T04:17:20.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.788338+0000", 2023-11-01T04:17:20.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100206936 2023-11-01T04:17:20.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.788353+0000", 2023-11-01T04:17:20.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5109000000000001e-05 2023-11-01T04:17:20.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.883595+0000", 2023-11-01T04:17:20.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.095241261999999993 2023-11-01T04:17:20.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:15.883646+0000", 2023-11-01T04:17:20.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1313999999999997e-05 2023-11-01T04:17:20.622 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.622 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.622 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 37/31 ECSubWrite(tid=20, reqid=client.4559.0:185, at_version=37'13, trim_to=0'0, roll_forward_to=36'11))", 2023-11-01T04:17:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.965368+0000", 2023-11-01T04:17:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 63.642797184999999, 2023-11-01T04:17:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10412571, 2023-11-01T04:17:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965368+0000", 2023-11-01T04:17:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965368+0000", 2023-11-01T04:17:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965369+0000", 2023-11-01T04:17:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4600000000000004e-07 2023-11-01T04:17:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965431+0000", 2023-11-01T04:17:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2322000000000004e-05 2023-11-01T04:17:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965465+0000", 2023-11-01T04:17:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4408e-05 2023-11-01T04:17:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965490+0000", 2023-11-01T04:17:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4272000000000001e-05 2023-11-01T04:17:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965716+0000", 2023-11-01T04:17:20.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022658800000000001 2023-11-01T04:17:20.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965742+0000", 2023-11-01T04:17:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5445000000000001e-05 2023-11-01T04:17:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.069410+0000", 2023-11-01T04:17:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10366858299999999 2023-11-01T04:17:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.069494+0000", 2023-11-01T04:17:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3446e-05 2023-11-01T04:17:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 47/31 ECSubWrite(tid=52, reqid=client.4559.0:350, at_version=47'43, trim_to=0'0, roll_forward_to=45'41))", 2023-11-01T04:17:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.037415+0000", 2023-11-01T04:17:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.570749812000003, 2023-11-01T04:17:20.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10750568200000001, 2023-11-01T04:17:20.635 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.635 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.635 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.037415+0000", 2023-11-01T04:17:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.037415+0000", 2023-11-01T04:17:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.037416+0000", 2023-11-01T04:17:20.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.17e-06 2023-11-01T04:17:20.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.037474+0000", 2023-11-01T04:17:20.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7584999999999998e-05 2023-11-01T04:17:20.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.037507+0000", 2023-11-01T04:17:20.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3358999999999997e-05 2023-11-01T04:17:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.137630+0000", 2023-11-01T04:17:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100122251 2023-11-01T04:17:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.142712+0000", 2023-11-01T04:17:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0050826220000000002 2023-11-01T04:17:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.142732+0000", 2023-11-01T04:17:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9619999999999998e-05 2023-11-01T04:17:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.144845+0000", 2023-11-01T04:17:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021129040000000001 2023-11-01T04:17:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.144921+0000", 2023-11-01T04:17:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6171e-05 2023-11-01T04:17:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.6s6 47/31 ECSubWrite(tid=43, reqid=client.4559.0:351, at_version=47'6, trim_to=0'0, roll_forward_to=36'4))", 2023-11-01T04:17:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.248407+0000", 2023-11-01T04:17:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.359758132000003, 2023-11-01T04:17:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202387496, 2023-11-01T04:17:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248407+0000", 2023-11-01T04:17:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248407+0000", 2023-11-01T04:17:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248408+0000", 2023-11-01T04:17:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.037e-06 2023-11-01T04:17:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248462+0000", 2023-11-01T04:17:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4497999999999999e-05 2023-11-01T04:17:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.248501+0000", 2023-11-01T04:17:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8423000000000001e-05 2023-11-01T04:17:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.348639+0000", 2023-11-01T04:17:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100138353 2023-11-01T04:17:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.448915+0000", 2023-11-01T04:17:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100276063 2023-11-01T04:17:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.448949+0000", 2023-11-01T04:17:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3482000000000002e-05 2023-11-01T04:17:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.450761+0000", 2023-11-01T04:17:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001811785 2023-11-01T04:17:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.450794+0000", 2023-11-01T04:17:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3855e-05 2023-11-01T04:17:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.as3 47/31 ECSubWrite(tid=77, reqid=client.4559.0:362, at_version=47'17, trim_to=0'0, roll_forward_to=44'15))", 2023-11-01T04:17:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:27.918704+0000", 2023-11-01T04:17:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 52.689460969000002, 2023-11-01T04:17:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10345299199999999, 2023-11-01T04:17:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.918704+0000", 2023-11-01T04:17:20.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.918704+0000", 2023-11-01T04:17:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.918705+0000", 2023-11-01T04:17:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5999999999999999e-07 2023-11-01T04:17:20.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.918936+0000", 2023-11-01T04:17:20.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023130000000000001 2023-11-01T04:17:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.918965+0000", 2023-11-01T04:17:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8745e-05 2023-11-01T04:17:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:27.918981+0000", 2023-11-01T04:17:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6529999999999999e-05 2023-11-01T04:17:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.019204+0000", 2023-11-01T04:17:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100222775 2023-11-01T04:17:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.019223+0000", 2023-11-01T04:17:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8958999999999999e-05 2023-11-01T04:17:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.022057+0000", 2023-11-01T04:17:20.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028342419999999998 2023-11-01T04:17:20.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.022157+0000", 2023-11-01T04:17:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9981000000000004e-05 2023-11-01T04:17:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.7s2 48/31 ECSubWrite(tid=46, reqid=client.4559.0:373, at_version=48'23, trim_to=0'0, roll_forward_to=46'20))", 2023-11-01T04:17:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:28.188861+0000", 2023-11-01T04:17:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 52.419304337, 2023-11-01T04:17:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20350378199999999, 2023-11-01T04:17:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.188861+0000", 2023-11-01T04:17:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.188861+0000", 2023-11-01T04:17:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.188862+0000", 2023-11-01T04:17:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2049999999999999e-06 2023-11-01T04:17:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.189188+0000", 2023-11-01T04:17:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00032597099999999999 2023-11-01T04:17:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.189217+0000", 2023-11-01T04:17:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9204999999999999e-05 2023-11-01T04:17:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.189237+0000", 2023-11-01T04:17:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.95e-05 2023-11-01T04:17:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.289496+0000", 2023-11-01T04:17:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100259104 2023-11-01T04:17:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.289526+0000", 2023-11-01T04:17:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9845e-05 2023-11-01T04:17:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.392321+0000", 2023-11-01T04:17:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10279562 2023-11-01T04:17:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.392364+0000", 2023-11-01T04:17:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3331999999999998e-05 2023-11-01T04:17:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.677 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.677 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.7s2 48/31 ECSubWrite(tid=47, reqid=client.4559.0:374, at_version=48'24, trim_to=0'0, roll_forward_to=46'20))", 2023-11-01T04:17:20.677 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:28.191530+0000", 2023-11-01T04:17:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 52.416635137999997, 2023-11-01T04:17:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.200865144, 2023-11-01T04:17:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.191530+0000", 2023-11-01T04:17:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.191530+0000", 2023-11-01T04:17:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.191531+0000", 2023-11-01T04:17:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9300000000000006e-07 2023-11-01T04:17:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.191574+0000", 2023-11-01T04:17:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3132e-05 2023-11-01T04:17:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.191588+0000", 2023-11-01T04:17:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4399000000000001e-05 2023-11-01T04:17:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.191596+0000", 2023-11-01T04:17:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3919999999999997e-06 2023-11-01T04:17:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.291060+0000", 2023-11-01T04:17:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.099463833000000001 2023-11-01T04:17:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.291088+0000", 2023-11-01T04:17:20.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8172999999999999e-05 2023-11-01T04:17:20.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.392372+0000", 2023-11-01T04:17:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101284208 2023-11-01T04:17:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.392395+0000", 2023-11-01T04:17:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3014e-05 2023-11-01T04:17:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.7s2 48/31 ECSubWrite(tid=48, reqid=client.4559.0:375, at_version=48'25, trim_to=0'0, roll_forward_to=46'20))", 2023-11-01T04:17:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:28.193820+0000", 2023-11-01T04:17:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 52.414345124, 2023-11-01T04:17:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19924911000000001, 2023-11-01T04:17:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.193820+0000", 2023-11-01T04:17:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.193820+0000", 2023-11-01T04:17:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.193821+0000", 2023-11-01T04:17:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0470000000000001e-06 2023-11-01T04:17:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.193842+0000", 2023-11-01T04:17:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0678000000000001e-05 2023-11-01T04:17:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.193858+0000", 2023-11-01T04:17:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6388e-05 2023-11-01T04:17:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.193867+0000", 2023-11-01T04:17:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.054e-06 2023-11-01T04:17:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.392049+0000", 2023-11-01T04:17:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.198182049 2023-11-01T04:17:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.392061+0000", 2023-11-01T04:17:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1687e-05 2023-11-01T04:17:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.392977+0000", 2023-11-01T04:17:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000916204 2023-11-01T04:17:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:28.393069+0000", 2023-11-01T04:17:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2003000000000007e-05 2023-11-01T04:17:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.699 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.6s6 54/31 ECSubWrite(tid=51, reqid=osd.14.0:51, at_version=54'8, trim_to=0'0, roll_forward_to=47'6))", 2023-11-01T04:17:20.699 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:34.105900+0000", 2023-11-01T04:17:20.699 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.502265315999999, 2023-11-01T04:17:20.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105133106, 2023-11-01T04:17:20.700 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.700 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.700 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.105900+0000", 2023-11-01T04:17:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.105900+0000", 2023-11-01T04:17:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.105900+0000", 2023-11-01T04:17:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4000000000000001e-07 2023-11-01T04:17:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.106017+0000", 2023-11-01T04:17:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117123 2023-11-01T04:17:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.106062+0000", 2023-11-01T04:17:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4536000000000002e-05 2023-11-01T04:17:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.106092+0000", 2023-11-01T04:17:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0375999999999999e-05 2023-11-01T04:17:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.206398+0000", 2023-11-01T04:17:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10030521100000001 2023-11-01T04:17:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.206425+0000", 2023-11-01T04:17:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7645999999999999e-05 2023-11-01T04:17:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.210955+0000", 2023-11-01T04:17:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045296499999999996 2023-11-01T04:17:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:34.211033+0000", 2023-11-01T04:17:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7923999999999995e-05 2023-11-01T04:17:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.1s2 57/31 ECSubWrite(tid=58, reqid=osd.6.0:58, at_version=57'34, trim_to=0'0, roll_forward_to=56'32))", 2023-11-01T04:17:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:37.442272+0000", 2023-11-01T04:17:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 43.165893486999998, 2023-11-01T04:17:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102805244, 2023-11-01T04:17:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.442272+0000", 2023-11-01T04:17:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.442272+0000", 2023-11-01T04:17:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.442272+0000", 2023-11-01T04:17:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4000000000000001e-07 2023-11-01T04:17:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.442326+0000", 2023-11-01T04:17:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3461999999999997e-05 2023-11-01T04:17:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.442363+0000", 2023-11-01T04:17:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7598999999999997e-05 2023-11-01T04:17:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.542467+0000", 2023-11-01T04:17:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100103435 2023-11-01T04:17:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.542650+0000", 2023-11-01T04:17:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000183499 2023-11-01T04:17:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.542677+0000", 2023-11-01T04:17:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6556000000000001e-05 2023-11-01T04:17:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.544983+0000", 2023-11-01T04:17:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023058789999999998 2023-11-01T04:17:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:37.545077+0000", 2023-11-01T04:17:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4173999999999997e-05 2023-11-01T04:17:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.7s2 58/31 ECSubWrite(tid=65, reqid=osd.14.0:65, at_version=58'38, trim_to=0'0, roll_forward_to=57'36))", 2023-11-01T04:17:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:38.452520+0000", 2023-11-01T04:17:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 42.155645307999997, 2023-11-01T04:17:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202505307, 2023-11-01T04:17:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.452520+0000", 2023-11-01T04:17:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.452520+0000", 2023-11-01T04:17:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.452520+0000", 2023-11-01T04:17:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1600000000000002e-07 2023-11-01T04:17:20.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.452553+0000", 2023-11-01T04:17:20.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3322999999999999e-05 2023-11-01T04:17:20.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.452581+0000", 2023-11-01T04:17:20.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7665999999999999e-05 2023-11-01T04:17:20.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.452597+0000", 2023-11-01T04:17:20.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6413e-05 2023-11-01T04:17:20.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.729 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.652927+0000", 2023-11-01T04:17:20.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20032907899999999 2023-11-01T04:17:20.730 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.730 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.730 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.652945+0000", 2023-11-01T04:17:20.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8896e-05 2023-11-01T04:17:20.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.731 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.654953+0000", 2023-11-01T04:17:20.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020074759999999998 2023-11-01T04:17:20.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:38.655025+0000", 2023-11-01T04:17:20.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2138000000000006e-05 2023-11-01T04:17:20.732 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.733 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.733 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.733 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.bs3 62/31 ECSubWrite(tid=70, reqid=client.4559.0:479, at_version=62'17, trim_to=0'0, roll_forward_to=45'15))", 2023-11-01T04:17:20.733 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:42.297728+0000", 2023-11-01T04:17:20.734 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.310437188999998, 2023-11-01T04:17:20.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10443308799999999, 2023-11-01T04:17:20.734 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.734 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.734 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.735 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.297728+0000", 2023-11-01T04:17:20.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.297728+0000", 2023-11-01T04:17:20.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.297729+0000", 2023-11-01T04:17:20.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1699999999999997e-07 2023-11-01T04:17:20.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.298270+0000", 2023-11-01T04:17:20.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00054102399999999998 2023-11-01T04:17:20.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.298303+0000", 2023-11-01T04:17:20.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3785999999999998e-05 2023-11-01T04:17:20.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.298331+0000", 2023-11-01T04:17:20.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7330000000000001e-05 2023-11-01T04:17:20.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.398610+0000", 2023-11-01T04:17:20.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100279405 2023-11-01T04:17:20.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.398627+0000", 2023-11-01T04:17:20.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6685999999999999e-05 2023-11-01T04:17:20.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.402088+0000", 2023-11-01T04:17:20.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034616550000000001 2023-11-01T04:17:20.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.402161+0000", 2023-11-01T04:17:20.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2484999999999995e-05 2023-11-01T04:17:20.743 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.743 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.744 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.744 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.as3 67/31 ECSubWrite(tid=131, reqid=client.4559.0:535, at_version=0'0, trim_to=0'0, roll_forward_to=67'30))", 2023-11-01T04:17:20.744 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.392875+0000", 2023-11-01T04:17:20.744 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 33.215290019000001, 2023-11-01T04:17:20.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20174935799999999, 2023-11-01T04:17:20.745 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.745 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.745 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.392875+0000", 2023-11-01T04:17:20.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.392875+0000", 2023-11-01T04:17:20.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.392878+0000", 2023-11-01T04:17:20.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.88e-06 2023-11-01T04:17:20.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.392907+0000", 2023-11-01T04:17:20.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8693000000000001e-05 2023-11-01T04:17:20.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.749 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.392954+0000", 2023-11-01T04:17:20.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7688999999999997e-05 2023-11-01T04:17:20.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.493051+0000", 2023-11-01T04:17:20.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10009689200000001 2023-11-01T04:17:20.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.593262+0000", 2023-11-01T04:17:20.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10021129400000001 2023-11-01T04:17:20.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.752 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.593281+0000", 2023-11-01T04:17:20.752 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8695999999999999e-05 2023-11-01T04:17:20.752 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.752 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.752 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.594577+0000", 2023-11-01T04:17:20.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012961680000000001 2023-11-01T04:17:20.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.753 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.754 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.594624+0000", 2023-11-01T04:17:20.754 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7046000000000003e-05 2023-11-01T04:17:20.754 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.754 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.754 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.755 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 67/31 ECSubWrite(tid=77, reqid=client.4559.0:537, at_version=67'58, trim_to=0'0, roll_forward_to=67'56))", 2023-11-01T04:17:20.755 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.705899+0000", 2023-11-01T04:17:20.755 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 32.902266079, 2023-11-01T04:17:20.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20538421000000001, 2023-11-01T04:17:20.756 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.756 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.756 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.756 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.757 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.705899+0000", 2023-11-01T04:17:20.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.758 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.705899+0000", 2023-11-01T04:17:20.758 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.758 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.758 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.759 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.705900+0000", 2023-11-01T04:17:20.759 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5200000000000001e-06 2023-11-01T04:17:20.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.759 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.759 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.760 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.706759+0000", 2023-11-01T04:17:20.760 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00085813999999999997 2023-11-01T04:17:20.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.760 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.760 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.761 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.706787+0000", 2023-11-01T04:17:20.761 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8385999999999999e-05 2023-11-01T04:17:20.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.761 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.761 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.761 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.806881+0000", 2023-11-01T04:17:20.762 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10009428200000001 2023-11-01T04:17:20.762 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.762 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.762 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.762 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.907197+0000", 2023-11-01T04:17:20.763 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100315683 2023-11-01T04:17:20.763 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.763 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.763 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.763 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.907230+0000", 2023-11-01T04:17:20.764 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3464999999999999e-05 2023-11-01T04:17:20.764 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.764 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.764 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.764 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.911219+0000", 2023-11-01T04:17:20.765 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039884150000000004 2023-11-01T04:17:20.765 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.765 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.765 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.765 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.911283+0000", 2023-11-01T04:17:20.765 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4319000000000005e-05 2023-11-01T04:17:20.766 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.766 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.766 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.766 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.766 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.767 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.7s2 68/31 ECSubWrite(tid=83, reqid=client.4559.0:546, at_version=68'44, trim_to=0'0, roll_forward_to=67'42))", 2023-11-01T04:17:20.767 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.893994+0000", 2023-11-01T04:17:20.767 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 32.714171172, 2023-11-01T04:17:20.767 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10444176500000001, 2023-11-01T04:17:20.767 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.768 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.768 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.768 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.768 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.768 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.893994+0000", 2023-11-01T04:17:20.768 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.769 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.769 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.769 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.893994+0000", 2023-11-01T04:17:20.769 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.770 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.770 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.770 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.893996+0000", 2023-11-01T04:17:20.770 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9769999999999999e-06 2023-11-01T04:17:20.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.771 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.771 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.771 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894238+0000", 2023-11-01T04:17:20.771 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024247900000000001 2023-11-01T04:17:20.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.772 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.772 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.772 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894281+0000", 2023-11-01T04:17:20.772 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2392999999999999e-05 2023-11-01T04:17:20.772 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.773 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.773 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.773 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.894303+0000", 2023-11-01T04:17:20.773 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2612e-05 2023-11-01T04:17:20.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.773 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.774 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.774 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.994602+0000", 2023-11-01T04:17:20.774 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100298312 2023-11-01T04:17:20.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.774 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.775 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.775 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.994621+0000", 2023-11-01T04:17:20.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.893e-05 2023-11-01T04:17:20.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.775 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.776 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.776 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.998363+0000", 2023-11-01T04:17:20.776 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037421260000000001 2023-11-01T04:17:20.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.776 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.776 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.777 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.998436+0000", 2023-11-01T04:17:20.777 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2935999999999994e-05 2023-11-01T04:17:20.777 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.777 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.777 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.777 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.778 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.778 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.7s2 73/72 ECSubWrite(tid=110, reqid=client.4559.0:613, at_version=0'0, trim_to=0'0, roll_forward_to=73'64))", 2023-11-01T04:17:20.778 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:54.547785+0000", 2023-11-01T04:17:20.778 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 26.060379953000002, 2023-11-01T04:17:20.778 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.200821892, 2023-11-01T04:17:20.779 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.779 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.779 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.779 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.779 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.779 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:54.547785+0000", 2023-11-01T04:17:20.780 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.780 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.780 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.780 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:54.547785+0000", 2023-11-01T04:17:20.781 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.781 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.781 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.781 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.781 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:54.547787+0000", 2023-11-01T04:17:20.782 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.159e-06 2023-11-01T04:17:20.782 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.782 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.782 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.782 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:54.547827+0000", 2023-11-01T04:17:20.782 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9539000000000002e-05 2023-11-01T04:17:20.783 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.783 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.783 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.783 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:54.547860+0000", 2023-11-01T04:17:20.783 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2898000000000002e-05 2023-11-01T04:17:20.783 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.784 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.784 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.784 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:54.648000+0000", 2023-11-01T04:17:20.784 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100140194 2023-11-01T04:17:20.784 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.785 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.785 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.785 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:54.648154+0000", 2023-11-01T04:17:20.785 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000153967 2023-11-01T04:17:20.785 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.786 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.786 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.786 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:54.648163+0000", 2023-11-01T04:17:20.786 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5670000000000001e-06 2023-11-01T04:17:20.786 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.787 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.787 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.787 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:54.748570+0000", 2023-11-01T04:17:20.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10040703300000001 2023-11-01T04:17:20.787 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.788 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:54.748607+0000", 2023-11-01T04:17:20.788 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6535000000000002e-05 2023-11-01T04:17:20.788 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.788 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.789 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.789 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.789 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.es1 74/73 ECSubWrite(tid=117, reqid=osd.11.0:117, at_version=74'104, trim_to=0'0, roll_forward_to=74'102))", 2023-11-01T04:17:20.789 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:55.722218+0000", 2023-11-01T04:17:20.790 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 24.885947045999998, 2023-11-01T04:17:20.790 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.103126671, 2023-11-01T04:17:20.790 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.790 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.790 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.790 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.791 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.791 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.722218+0000", 2023-11-01T04:17:20.791 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.791 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.792 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.722218+0000", 2023-11-01T04:17:20.792 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.792 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.793 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.722220+0000", 2023-11-01T04:17:20.793 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.73e-06 2023-11-01T04:17:20.793 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.793 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.793 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.794 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.722283+0000", 2023-11-01T04:17:20.794 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3280000000000004e-05 2023-11-01T04:17:20.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.794 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.794 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.795 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.722314+0000", 2023-11-01T04:17:20.795 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0676000000000002e-05 2023-11-01T04:17:20.795 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.795 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.795 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.795 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.822422+0000", 2023-11-01T04:17:20.796 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100108211 2023-11-01T04:17:20.796 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.796 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.796 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.796 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.822720+0000", 2023-11-01T04:17:20.797 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029780300000000002 2023-11-01T04:17:20.797 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.797 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.797 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.797 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.822766+0000", 2023-11-01T04:17:20.797 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5967999999999998e-05 2023-11-01T04:17:20.798 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.798 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.798 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.798 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.825244+0000", 2023-11-01T04:17:20.798 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002478821 2023-11-01T04:17:20.799 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.799 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.799 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.799 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:55.825345+0000", 2023-11-01T04:17:20.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000100182 2023-11-01T04:17:20.800 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.800 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.800 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.800 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.800 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.800 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.1s2 82/78 ECSubWrite(tid=113, reqid=client.4559.0:736, at_version=82'67, trim_to=0'0, roll_forward_to=0'0))", 2023-11-01T04:17:20.801 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:03.743557+0000", 2023-11-01T04:17:20.801 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 16.864608368999999, 2023-11-01T04:17:20.801 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10618704399999999, 2023-11-01T04:17:20.801 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.801 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.801 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.802 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.802 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.743557+0000", 2023-11-01T04:17:20.802 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.743557+0000", 2023-11-01T04:17:20.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.803 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.804 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.804 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.743563+0000", 2023-11-01T04:17:20.804 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3890000000000002e-06 2023-11-01T04:17:20.804 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.804 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.804 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.805 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.743605+0000", 2023-11-01T04:17:20.805 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1875000000000001e-05 2023-11-01T04:17:20.805 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.805 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.805 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.805 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.743631+0000", 2023-11-01T04:17:20.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5989000000000001e-05 2023-11-01T04:17:20.806 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.806 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.806 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.806 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.743649+0000", 2023-11-01T04:17:20.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8332999999999999e-05 2023-11-01T04:17:20.807 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.807 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.807 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.807 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.747898+0000", 2023-11-01T04:17:20.807 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042484929999999999 2023-11-01T04:17:20.807 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.808 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.808 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.808 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.747916+0000", 2023-11-01T04:17:20.808 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8586000000000001e-05 2023-11-01T04:17:20.808 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.808 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.809 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.809 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.849689+0000", 2023-11-01T04:17:20.809 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.101772287 2023-11-01T04:17:20.809 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.809 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.810 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.810 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:03.849744+0000", 2023-11-01T04:17:20.810 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5092e-05 2023-11-01T04:17:20.810 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.811 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.811 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.811 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.811 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.811 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "pg_update_log_missing(3.es1 epoch 93/73 rep_tid 163 entries 93'154 (0'0) error 3:7f8a06cd:::smithi19039307-49:head by client.4559.0:846 0.000000 -2 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 90'151)", 2023-11-01T04:17:20.811 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:14.742268+0000", 2023-11-01T04:17:20.812 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.8658974859999997, 2023-11-01T04:17:20.812 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.174626578, 2023-11-01T04:17:20.812 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.812 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.812 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.812 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.813 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:14.742268+0000", 2023-11-01T04:17:20.813 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.813 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.814 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:14.742268+0000", 2023-11-01T04:17:20.814 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.814 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.814 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.814 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.815 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:14.742268+0000", 2023-11-01T04:17:20.815 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8999999999999996e-07 2023-11-01T04:17:20.815 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.815 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.815 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.815 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:14.742282+0000", 2023-11-01T04:17:20.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3957e-05 2023-11-01T04:17:20.816 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.816 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.816 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.816 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:14.742330+0000", 2023-11-01T04:17:20.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7478000000000002e-05 2023-11-01T04:17:20.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.817 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:14.742365+0000", 2023-11-01T04:17:20.817 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5892000000000001e-05 2023-11-01T04:17:20.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.818 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.818 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.818 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:14.915478+0000", 2023-11-01T04:17:20.818 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.173112823 2023-11-01T04:17:20.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:14.916894+0000", 2023-11-01T04:17:20.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001415838 2023-11-01T04:17:20.819 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.819 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.820 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.820 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.820 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpWrite(3.6s6 97/68 ECSubWrite(tid=155, reqid=client.4559.0:916, at_version=97'24, trim_to=0'0, roll_forward_to=90'22))", 2023-11-01T04:17:20.820 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:19.081045+0000", 2023-11-01T04:17:20.820 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.527120357, 2023-11-01T04:17:20.821 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10380891, 2023-11-01T04:17:20.821 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:20.821 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:20.821 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:20.821 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.821 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:20.822 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.081045+0000", 2023-11-01T04:17:20.822 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.822 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.822 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.822 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:20.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.081045+0000", 2023-11-01T04:17:20.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:20.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.823 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:20.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.081045+0000", 2023-11-01T04:17:20.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8500000000000002e-07 2023-11-01T04:17:20.824 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.824 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.824 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:20.824 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.082021+0000", 2023-11-01T04:17:20.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00097552200000000004 2023-11-01T04:17:20.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.825 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.825 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:20.825 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.082039+0000", 2023-11-01T04:17:20.825 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8363999999999999e-05 2023-11-01T04:17:20.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.826 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.826 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:20.826 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.082050+0000", 2023-11-01T04:17:20.826 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0842000000000001e-05 2023-11-01T04:17:20.826 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.827 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.827 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:20.827 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.183291+0000", 2023-11-01T04:17:20.827 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10124083 2023-11-01T04:17:20.827 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.827 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:20.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.183309+0000", 2023-11-01T04:17:20.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8357000000000001e-05 2023-11-01T04:17:20.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:20.829 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.184826+0000", 2023-11-01T04:17:20.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001516938 2023-11-01T04:17:20.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:20.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:20.829 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:20.830 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.184854+0000", 2023-11-01T04:17:20.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7772000000000001e-05 2023-11-01T04:17:20.830 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.830 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.830 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.831 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:20.831 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:20.831 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:20.832 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2023-11-01T04:17:20.833 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:20.830+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:20.933 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:20.930+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:20.962 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:20.963 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:17:20.963 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2023-11-01T04:17:20.963 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:20.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2023-11-01T04:17:21.034 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:17:21.032+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:17:21.077 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-11-01T04:17:21.090 INFO:tasks.rados.rados.0.smithi190.stdout:382: done (1 left) 2023-11-01T04:17:21.091 INFO:tasks.rados.rados.0.smithi190.stdout:383: done (0 left) 2023-11-01T04:17:21.091 INFO:tasks.rados.rados.0.smithi190.stdout:384: rollback oid 41 current snap is 36 2023-11-01T04:17:21.091 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 41 to 34 2023-11-01T04:17:21.091 INFO:tasks.rados.rados.0.smithi190.stdout:385: rollback oid 18 current snap is 36 2023-11-01T04:17:21.091 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 18 to 28 2023-11-01T04:17:21.092 INFO:tasks.rados.rados.0.smithi190.stdout:386: rmattr oid 23 current snap is 36 2023-11-01T04:17:21.092 INFO:tasks.rados.rados.0.smithi190.stdout:386: done (2 left) 2023-11-01T04:17:21.092 INFO:tasks.rados.rados.0.smithi190.stdout:387: copy_from oid 43 from oid 10 current snap is 36 2023-11-01T04:17:21.092 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 19 current snap is 36 2023-11-01T04:17:21.092 INFO:tasks.rados.rados.0.smithi190.stdout:388: seq_num 139 ranges {0=786432,786432=688128,1474560=475136,1949696=393216} 2023-11-01T04:17:21.102 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:21.102 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2023-11-01T04:17:21.102 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2023-11-01T04:17:21.103 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2023-11-01T04:17:21.103 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:21.103 INFO:tasks.rados.rados.0.smithi190.stdout:388: writing smithi19039307-19 from 0 to 786432 tid 1 2023-11-01T04:17:21.115 INFO:tasks.rados.rados.0.smithi190.stdout:388: writing smithi19039307-19 from 786432 to 1474560 tid 2 2023-11-01T04:17:21.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2023-11-01T04:17:21.122 INFO:tasks.rados.rados.0.smithi190.stdout:388: writing smithi19039307-19 from 1474560 to 1949696 tid 3 2023-11-01T04:17:21.129 INFO:tasks.rados.rados.0.smithi190.stdout:388: writing smithi19039307-19 from 1949696 to 2342912 tid 4 2023-11-01T04:17:21.129 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 13 current snap is 36 2023-11-01T04:17:21.129 INFO:tasks.rados.rados.0.smithi190.stdout:389: seq_num 140 ranges {114688=802816,917504=425984,1343488=720896,2064384=409600} 2023-11-01T04:17:21.143 INFO:tasks.rados.rados.0.smithi190.stdout:389: writing smithi19039307-13 from 114688 to 917504 tid 1 2023-11-01T04:17:21.149 INFO:tasks.rados.rados.0.smithi190.stdout:389: writing smithi19039307-13 from 917504 to 1343488 tid 2 2023-11-01T04:17:21.158 INFO:tasks.rados.rados.0.smithi190.stdout:389: writing smithi19039307-13 from 1343488 to 2064384 tid 3 2023-11-01T04:17:21.162 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:17:21.160+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:17:21.162 INFO:tasks.rados.rados.0.smithi190.stdout:389: writing smithi19039307-13 from 2064384 to 2473984 tid 4 2023-11-01T04:17:21.163 INFO:tasks.rados.rados.0.smithi190.stdout:390: setattr oid 26 current snap is 36 2023-11-01T04:17:21.166 INFO:tasks.rados.rados.0.smithi190.stdout:391: snap_create 2023-11-01T04:17:21.235 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:21.249 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2023-11-01T04:17:21.249 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2023-11-01T04:17:21.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2023-11-01T04:17:21.249 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2023-11-01T04:17:21.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.250 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:17:21.250 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138761+0000", 2023-11-01T04:17:21.250 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 76.106939613999998, 2023-11-01T04:17:21.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61681865999999996, 2023-11-01T04:17:21.250 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.251 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:21.251 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.251 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:17:21.251 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:17:21.251 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 114 2023-11-01T04:17:21.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.252 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138761+0000", 2023-11-01T04:17:21.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138761+0000", 2023-11-01T04:17:21.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138762+0000", 2023-11-01T04:17:21.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4300000000000003e-07 2023-11-01T04:17:21.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138768+0000", 2023-11-01T04:17:21.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7729999999999996e-06 2023-11-01T04:17:21.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138777+0000", 2023-11-01T04:17:21.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1929999999999999e-06 2023-11-01T04:17:21.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138784+0000", 2023-11-01T04:17:21.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3830000000000003e-06 2023-11-01T04:17:21.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.248094+0000", 2023-11-01T04:17:21.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10930951699999999 2023-11-01T04:17:21.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.248210+0000", 2023-11-01T04:17:21.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011579299999999999 2023-11-01T04:17:21.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:17:21.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.248407+0000", 2023-11-01T04:17:21.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019723000000000001 2023-11-01T04:17:21.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:17:21.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352490+0000", 2023-11-01T04:17:21.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104083393 2023-11-01T04:17:21.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:17:21.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755390+0000", 2023-11-01T04:17:21.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40290009500000001 2023-11-01T04:17:21.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:21.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755474+0000", 2023-11-01T04:17:21.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3323000000000002e-05 2023-11-01T04:17:21.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755580+0000", 2023-11-01T04:17:21.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010631700000000001 2023-11-01T04:17:21.265 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.265 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.265 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.265 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:17:21.266 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138795+0000", 2023-11-01T04:17:21.266 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 76.106905456000007, 2023-11-01T04:17:21.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61707305000000001, 2023-11-01T04:17:21.266 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.266 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:21.267 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.267 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:17:21.267 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:17:21.267 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 115 2023-11-01T04:17:21.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.268 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138795+0000", 2023-11-01T04:17:21.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138795+0000", 2023-11-01T04:17:21.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138796+0000", 2023-11-01T04:17:21.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3300000000000002e-07 2023-11-01T04:17:21.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138931+0000", 2023-11-01T04:17:21.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013525000000000001 2023-11-01T04:17:21.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138947+0000", 2023-11-01T04:17:21.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5478999999999999e-05 2023-11-01T04:17:21.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138956+0000", 2023-11-01T04:17:21.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1530000000000003e-06 2023-11-01T04:17:21.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.249005+0000", 2023-11-01T04:17:21.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.110048914 2023-11-01T04:17:21.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.249150+0000", 2023-11-01T04:17:21.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014491600000000001 2023-11-01T04:17:21.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:17:21.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.249351+0000", 2023-11-01T04:17:21.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000201516 2023-11-01T04:17:21.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:17:21.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352738+0000", 2023-11-01T04:17:21.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10338632 2023-11-01T04:17:21.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:17:21.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755668+0000", 2023-11-01T04:17:21.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40293035700000002 2023-11-01T04:17:21.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:21.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755765+0000", 2023-11-01T04:17:21.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7081000000000001e-05 2023-11-01T04:17:21.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.755868+0000", 2023-11-01T04:17:21.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010343100000000001 2023-11-01T04:17:21.279 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:17:21.280 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.138971+0000", 2023-11-01T04:17:21.281 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 76.106729917999999, 2023-11-01T04:17:21.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71733164100000002, 2023-11-01T04:17:21.281 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.281 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:21.281 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.282 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:17:21.282 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:17:21.282 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 116 2023-11-01T04:17:21.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.282 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138971+0000", 2023-11-01T04:17:21.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138971+0000", 2023-11-01T04:17:21.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138972+0000", 2023-11-01T04:17:21.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4000000000000001e-07 2023-11-01T04:17:21.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138988+0000", 2023-11-01T04:17:21.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6354999999999998e-05 2023-11-01T04:17:21.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.138998+0000", 2023-11-01T04:17:21.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0285999999999999e-05 2023-11-01T04:17:21.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139010+0000", 2023-11-01T04:17:21.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1807000000000001e-05 2023-11-01T04:17:21.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.350520+0000", 2023-11-01T04:17:21.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21151004200000001 2023-11-01T04:17:21.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.350668+0000", 2023-11-01T04:17:21.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014824900000000001 2023-11-01T04:17:21.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:17:21.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.350873+0000", 2023-11-01T04:17:21.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020507899999999999 2023-11-01T04:17:21.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:17:21.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.352744+0000", 2023-11-01T04:17:21.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001870741 2023-11-01T04:17:21.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:17:21.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856134+0000", 2023-11-01T04:17:21.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50339015399999998 2023-11-01T04:17:21.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:21.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856235+0000", 2023-11-01T04:17:21.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010087200000000001 2023-11-01T04:17:21.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856303+0000", 2023-11-01T04:17:21.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7316000000000004e-05 2023-11-01T04:17:21.294 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.294 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.295 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.295 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:17:21.295 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:05.139035+0000", 2023-11-01T04:17:21.296 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 76.106665931999999, 2023-11-01T04:17:21.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71770837499999995, 2023-11-01T04:17:21.296 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.296 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:21.296 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.296 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:17:21.297 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:17:21.297 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 117 2023-11-01T04:17:21.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.297 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139035+0000", 2023-11-01T04:17:21.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139035+0000", 2023-11-01T04:17:21.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139036+0000", 2023-11-01T04:17:21.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7999999999999995e-07 2023-11-01T04:17:21.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139042+0000", 2023-11-01T04:17:21.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9499999999999998e-06 2023-11-01T04:17:21.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139051+0000", 2023-11-01T04:17:21.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7489999999999993e-06 2023-11-01T04:17:21.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.139059+0000", 2023-11-01T04:17:21.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8159999999999994e-06 2023-11-01T04:17:21.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.351436+0000", 2023-11-01T04:17:21.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21237641400000001 2023-11-01T04:17:21.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.351583+0000", 2023-11-01T04:17:21.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000147186 2023-11-01T04:17:21.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:17:21.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.351772+0000", 2023-11-01T04:17:21.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018879300000000001 2023-11-01T04:17:21.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:17:21.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.553963+0000", 2023-11-01T04:17:21.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20219130599999999 2023-11-01T04:17:21.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:17:21.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856458+0000", 2023-11-01T04:17:21.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30249546300000002 2023-11-01T04:17:21.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:21.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856632+0000", 2023-11-01T04:17:21.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000173611 2023-11-01T04:17:21.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:05.856743+0000", 2023-11-01T04:17:21.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111407 2023-11-01T04:17:21.309 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.309 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.309 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.310 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4241.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2023-11-01T04:17:21.310 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:06.622843+0000", 2023-11-01T04:17:21.310 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 74.622857515000007, 2023-11-01T04:17:21.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70733220600000002, 2023-11-01T04:17:21.311 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.311 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:21.311 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.311 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4241", 2023-11-01T04:17:21.311 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/1674488683", 2023-11-01T04:17:21.311 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 141 2023-11-01T04:17:21.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.312 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622843+0000", 2023-11-01T04:17:21.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622843+0000", 2023-11-01T04:17:21.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622844+0000", 2023-11-01T04:17:21.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7400000000000002e-07 2023-11-01T04:17:21.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622850+0000", 2023-11-01T04:17:21.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4330000000000003e-06 2023-11-01T04:17:21.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.622859+0000", 2023-11-01T04:17:21.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5829999999999996e-06 2023-11-01T04:17:21.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.722958+0000", 2023-11-01T04:17:21.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100098824 2023-11-01T04:17:21.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.835872+0000", 2023-11-01T04:17:21.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.112913586 2023-11-01T04:17:21.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.836107+0000", 2023-11-01T04:17:21.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023507399999999999 2023-11-01T04:17:21.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 15", 2023-11-01T04:17:21.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.836474+0000", 2023-11-01T04:17:21.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036769800000000002 2023-11-01T04:17:21.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2023-11-01T04:17:21.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:06.837993+0000", 2023-11-01T04:17:21.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001518192 2023-11-01T04:17:21.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2023-11-01T04:17:21.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.329913+0000", 2023-11-01T04:17:21.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.49192055499999998 2023-11-01T04:17:21.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:21.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.330067+0000", 2023-11-01T04:17:21.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000154012 2023-11-01T04:17:21.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:07.330176+0000", 2023-11-01T04:17:21.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010847499999999999 2023-11-01T04:17:21.323 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.323 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.323 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.324 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:207 3.2s0 3:42ba522a:::smithi19039307-20:head [copy-from ver 4] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:21.324 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.965579+0000", 2023-11-01T04:17:21.324 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 64.280121519999994, 2023-11-01T04:17:21.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62034388299999998, 2023-11-01T04:17:21.325 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.325 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:21.325 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.325 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:21.325 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:21.325 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 207 2023-11-01T04:17:21.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.326 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965579+0000", 2023-11-01T04:17:21.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965579+0000", 2023-11-01T04:17:21.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965581+0000", 2023-11-01T04:17:21.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1969999999999999e-06 2023-11-01T04:17:21.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965617+0000", 2023-11-01T04:17:21.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6100999999999999e-05 2023-11-01T04:17:21.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965640+0000", 2023-11-01T04:17:21.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3566000000000001e-05 2023-11-01T04:17:21.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.965657+0000", 2023-11-01T04:17:21.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6413e-05 2023-11-01T04:17:21.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.066185+0000", 2023-11-01T04:17:21.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100528453 2023-11-01T04:17:21.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.066331+0000", 2023-11-01T04:17:21.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014583099999999999 2023-11-01T04:17:21.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:21.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.381227+0000", 2023-11-01T04:17:21.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31489625199999999 2023-11-01T04:17:21.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:21.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.384145+0000", 2023-11-01T04:17:21.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029175310000000001 2023-11-01T04:17:21.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:21.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.585865+0000", 2023-11-01T04:17:21.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.201720436 2023-11-01T04:17:21.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.585923+0000", 2023-11-01T04:17:21.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8103000000000003e-05 2023-11-01T04:17:21.336 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.336 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.336 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.337 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:211 3.3s0 3:ca2966a4:::smithi19039307-37:head [read 0~606208,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:21.337 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.966567+0000", 2023-11-01T04:17:21.337 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 64.279134335999998, 2023-11-01T04:17:21.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61304655900000005, 2023-11-01T04:17:21.338 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.338 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:21.338 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.338 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:21.339 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:21.339 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 211 2023-11-01T04:17:21.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.339 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966567+0000", 2023-11-01T04:17:21.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966567+0000", 2023-11-01T04:17:21.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966568+0000", 2023-11-01T04:17:21.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2529999999999999e-06 2023-11-01T04:17:21.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966575+0000", 2023-11-01T04:17:21.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4030000000000001e-06 2023-11-01T04:17:21.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966595+0000", 2023-11-01T04:17:21.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0208999999999999e-05 2023-11-01T04:17:21.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.966611+0000", 2023-11-01T04:17:21.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6082999999999998e-05 2023-11-01T04:17:21.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.072213+0000", 2023-11-01T04:17:21.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.105601509 2023-11-01T04:17:21.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.072432+0000", 2023-11-01T04:17:21.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000218724 2023-11-01T04:17:21.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.579613+0000", 2023-11-01T04:17:21.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50718137799999996 2023-11-01T04:17:21.347 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.347 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.348 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.348 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:212 3.3s0 3:c87735c0:::smithi19039307-50:head [create,setxattr XNeZS[lOPtVca\\Pa[\\rdfbinb[bcjhtYSpmcafZm (1928) in=40b,setxattr TfflYYqPnps]lo^qSlp___TitNdd`]kaPTVOTcsh (1648) in=40b,setxattr TNb_onpY[j_dkhfmnWtm]Yktg]]RNtbQe[OUTNlX (770) in=40b,setxattr ZcgnkhY\\orTYaZh]PPldj`SNq`p\\cX`k]h\\bZY_Y (700) in=40b,setxattr gPQVnqhpsVhW_kq`QnafXrlRZtZ\\YnRftgQTishR (595) in=40b,setxattr U`XqYi[WhoqnpSPRc\\RhnppZp^qY\\qUfjpqs[r^a (1892) in=40b,setxattr cTmhd`pWgrRf_VfVgOtgeOP^ShP]lsOtpoTj[\\gl (418) in=40b,setxattr aa_taYTfslg`lRRrOkkraN\\XiW^V\\ljl]NqXin`a (314) in=40b,setxattr ibl]`igtgVThY^Qse\\okYYUtTmolX\\W]p]Q[P\\il (1403) in=40b,setxattr \\ZabSO[UUqeZa`hddNh[ZjaRRgUZoQtkn\\QdsX]W (1541) in=40b,setxattr jUkQUPTNbo`rZNmYlUmhb]ojc]aeheoVRfp]bPWf (5052) in=40b,setxattr Ofj`oQ^cR]da^SPsTaXmmPse\\OZ`VSqQ]U`efsZl (419) in=40b,setxattr a[cdkWos\\OrUVPciUaa^_^`lWVTScffiWkd\\[kgf (1261) in=40b,setxattr jXb]\\WhbatdZ[qTTiZO]kPd^`l[kajOYiQ[iqSYc (244) in=40b,setxattr \\im]jQhbanUn\\_^h`jaU]W`ReaZN_je[]kgg (1578) in=36b,setxattr ZbosoRORS^_de]rN^_daOiha\\Q\\eRae^[sQUjXsV (9120) in=40b,setxattr gcn]XpZUkOZr\\pkrVVZpYiR\\alh_PaZa_S[PYcn] (384) in=40b,setxattr \\^qUOaNXSTjtmafhfbo\\gseTTkV`mgnacYq^bkcc (34) in=40b,setxattr iUa^mOQ]TqjVSUU_nXfY]f`NjljiiaYbrm[o]aes (1611) in=40b,setxattr tXaQPQbc^aiV[^reVOPpib^caoWf^XgqilO^_g[k (1016) in=40b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:21.348 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:16.970354+0000", 2023-11-01T04:17:21.349 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 64.275346726999999, 2023-11-01T04:17:21.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91502122100000005, 2023-11-01T04:17:21.349 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.349 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:21.349 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.350 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:21.350 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:21.350 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 212 2023-11-01T04:17:21.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.350 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970354+0000", 2023-11-01T04:17:21.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970354+0000", 2023-11-01T04:17:21.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970356+0000", 2023-11-01T04:17:21.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3070000000000001e-06 2023-11-01T04:17:21.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970468+0000", 2023-11-01T04:17:21.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111496 2023-11-01T04:17:21.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970492+0000", 2023-11-01T04:17:21.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4096000000000001e-05 2023-11-01T04:17:21.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:16.970511+0000", 2023-11-01T04:17:21.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9069e-05 2023-11-01T04:17:21.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.579670+0000", 2023-11-01T04:17:21.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60915843300000005 2023-11-01T04:17:21.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.579779+0000", 2023-11-01T04:17:21.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000109417 2023-11-01T04:17:21.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:21.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.580741+0000", 2023-11-01T04:17:21.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00096157299999999996 2023-11-01T04:17:21.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:21.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.582542+0000", 2023-11-01T04:17:21.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018016130000000001 2023-11-01T04:17:21.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:21.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.885257+0000", 2023-11-01T04:17:21.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30271534300000003 2023-11-01T04:17:21.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:17.885375+0000", 2023-11-01T04:17:21.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117874 2023-11-01T04:17:21.362 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.362 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.362 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.363 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:496 3.2s0 3:4174a5c9:::smithi19039307-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2023-11-01T04:17:21.363 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:42.354747+0000", 2023-11-01T04:17:21.363 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 38.890954247000003, 2023-11-01T04:17:21.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61943324300000002, 2023-11-01T04:17:21.363 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.364 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:21.364 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.364 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:21.364 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:21.364 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 496 2023-11-01T04:17:21.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.365 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354747+0000", 2023-11-01T04:17:21.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.366 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.366 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354747+0000", 2023-11-01T04:17:21.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354747+0000", 2023-11-01T04:17:21.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0299999999999998e-07 2023-11-01T04:17:21.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354751+0000", 2023-11-01T04:17:21.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.326e-06 2023-11-01T04:17:21.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354763+0000", 2023-11-01T04:17:21.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2716e-05 2023-11-01T04:17:21.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.354772+0000", 2023-11-01T04:17:21.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4570000000000005e-06 2023-11-01T04:17:21.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.357535+0000", 2023-11-01T04:17:21.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002763148 2023-11-01T04:17:21.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:21.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.357552+0000", 2023-11-01T04:17:21.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6849000000000001e-05 2023-11-01T04:17:21.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.568063+0000", 2023-11-01T04:17:21.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.210510788 2023-11-01T04:17:21.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:21.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.568200+0000", 2023-11-01T04:17:21.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013761000000000001 2023-11-01T04:17:21.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.670978+0000", 2023-11-01T04:17:21.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.102777572 2023-11-01T04:17:21.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.671121+0000", 2023-11-01T04:17:21.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014276299999999999 2023-11-01T04:17:21.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.376 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:42.974180+0000", 2023-11-01T04:17:21.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.303059211 2023-11-01T04:17:21.376 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.376 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.376 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.377 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:552 3.2s0 3:42f00163:::smithi19039307-2:head [copy-from ver 12] snapc 1e=[1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:21.377 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.820369+0000", 2023-11-01T04:17:21.377 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 33.425332003000001, 2023-11-01T04:17:21.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89219467399999997, 2023-11-01T04:17:21.378 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.378 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:21.378 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.378 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:21.378 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:21.378 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 552 2023-11-01T04:17:21.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.379 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820369+0000", 2023-11-01T04:17:21.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820369+0000", 2023-11-01T04:17:21.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820371+0000", 2023-11-01T04:17:21.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.266e-06 2023-11-01T04:17:21.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.382 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820383+0000", 2023-11-01T04:17:21.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2023e-05 2023-11-01T04:17:21.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820415+0000", 2023-11-01T04:17:21.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2255999999999997e-05 2023-11-01T04:17:21.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820431+0000", 2023-11-01T04:17:21.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5823000000000001e-05 2023-11-01T04:17:21.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.079379+0000", 2023-11-01T04:17:21.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25894801099999998 2023-11-01T04:17:21.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.079550+0000", 2023-11-01T04:17:21.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017098600000000001 2023-11-01T04:17:21.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:21.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.494335+0000", 2023-11-01T04:17:21.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41478489200000002 2023-11-01T04:17:21.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.387 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.387 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:21.387 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.498143+0000", 2023-11-01T04:17:21.387 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038076920000000001 2023-11-01T04:17:21.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.388 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:21.388 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.712428+0000", 2023-11-01T04:17:21.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21428475699999999 2023-11-01T04:17:21.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.389 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.389 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.712564+0000", 2023-11-01T04:17:21.389 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013596800000000001 2023-11-01T04:17:21.389 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.389 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.390 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.390 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.390 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:553 3.2s0 3:42f00163:::smithi19039307-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e68)", 2023-11-01T04:17:21.390 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:47.820458+0000", 2023-11-01T04:17:21.390 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 33.425242648000001, 2023-11-01T04:17:21.391 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.67673522200000003, 2023-11-01T04:17:21.391 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.391 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:21.391 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.391 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:21.392 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:21.392 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 553 2023-11-01T04:17:21.392 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.392 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.392 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.393 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.393 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820458+0000", 2023-11-01T04:17:21.393 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.393 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.393 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.394 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820458+0000", 2023-11-01T04:17:21.394 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.394 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.394 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.395 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820459+0000", 2023-11-01T04:17:21.395 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8299999999999995e-07 2023-11-01T04:17:21.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.395 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.395 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820463+0000", 2023-11-01T04:17:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9369999999999997e-06 2023-11-01T04:17:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.396 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.397 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.820475+0000", 2023-11-01T04:17:21.397 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2026e-05 2023-11-01T04:17:21.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.397 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:47.920578+0000", 2023-11-01T04:17:21.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10010329 2023-11-01T04:17:21.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.283349+0000", 2023-11-01T04:17:21.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36277072599999999 2023-11-01T04:17:21.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for blocked object", 2023-11-01T04:17:21.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.283446+0000", 2023-11-01T04:17:21.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7083999999999994e-05 2023-11-01T04:17:21.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.496890+0000", 2023-11-01T04:17:21.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21344336899999999 2023-11-01T04:17:21.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.497036+0000", 2023-11-01T04:17:21.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000146324 2023-11-01T04:17:21.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.402 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.402 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:48.497193+0000", 2023-11-01T04:17:21.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015758300000000001 2023-11-01T04:17:21.402 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.403 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.403 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.403 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.403 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.403 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:567 3.2s0 3:42f00163:::smithi19039307-2:head [read 0~557056,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2023-11-01T04:17:21.404 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:49.511529+0000", 2023-11-01T04:17:21.404 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 31.734172023999999, 2023-11-01T04:17:21.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72495039999999999, 2023-11-01T04:17:21.404 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.404 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:21.404 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.405 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:21.405 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:21.405 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 567 2023-11-01T04:17:21.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.405 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.406 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.406 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.406 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511529+0000", 2023-11-01T04:17:21.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.407 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.407 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511529+0000", 2023-11-01T04:17:21.407 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.408 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511530+0000", 2023-11-01T04:17:21.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2759999999999999e-06 2023-11-01T04:17:21.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.408 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.409 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511537+0000", 2023-11-01T04:17:21.409 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2899999999999997e-06 2023-11-01T04:17:21.409 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.409 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.409 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.410 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511557+0000", 2023-11-01T04:17:21.410 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9443e-05 2023-11-01T04:17:21.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.410 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.411 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.511577+0000", 2023-11-01T04:17:21.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9673e-05 2023-11-01T04:17:21.411 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.411 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.411 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.411 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.512998+0000", 2023-11-01T04:17:21.412 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014214360000000001 2023-11-01T04:17:21.412 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.412 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.412 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.412 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.513127+0000", 2023-11-01T04:17:21.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012906500000000001 2023-11-01T04:17:21.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.236479+0000", 2023-11-01T04:17:21.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72335221699999996 2023-11-01T04:17:21.414 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.414 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.414 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.414 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDECSubOpReadReply(3.2s0 69/31 ECSubReadReply(tid=322, attrs_read=0))", 2023-11-01T04:17:21.415 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:49.514831+0000", 2023-11-01T04:17:21.415 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 31.730869599999998, 2023-11-01T04:17:21.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72027060200000004, 2023-11-01T04:17:21.415 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.415 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2023-11-01T04:17:21.415 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.416 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.416 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.514831+0000", 2023-11-01T04:17:21.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.416 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.514831+0000", 2023-11-01T04:17:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.514831+0000", 2023-11-01T04:17:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.514831+0000", 2023-11-01T04:17:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.614593+0000", 2023-11-01T04:17:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.099762016999999995 2023-11-01T04:17:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.714692+0000", 2023-11-01T04:17:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10009886599999999 2023-11-01T04:17:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.235030+0000", 2023-11-01T04:17:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52033755299999995 2023-11-01T04:17:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.235102+0000", 2023-11-01T04:17:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2166e-05 2023-11-01T04:17:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.423 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:571 3.2s0 3:4a13469b:::smithi19039307-24:head [setxattr _header (57) in=7b] snapc 22=[22,1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2023-11-01T04:17:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:16:49.541372+0000", 2023-11-01T04:17:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 31.704329253000001, 2023-11-01T04:17:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98667972400000004, 2023-11-01T04:17:21.424 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:21.425 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 571 2023-11-01T04:17:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541372+0000", 2023-11-01T04:17:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541372+0000", 2023-11-01T04:17:21.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541373+0000", 2023-11-01T04:17:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8300000000000001e-06 2023-11-01T04:17:21.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541386+0000", 2023-11-01T04:17:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2146e-05 2023-11-01T04:17:21.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541410+0000", 2023-11-01T04:17:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4672999999999999e-05 2023-11-01T04:17:21.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:49.541427+0000", 2023-11-01T04:17:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6796e-05 2023-11-01T04:17:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.431 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.236577+0000", 2023-11-01T04:17:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.69514989800000004 2023-11-01T04:17:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.236756+0000", 2023-11-01T04:17:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000179211 2023-11-01T04:17:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.237530+0000", 2023-11-01T04:17:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077358099999999996 2023-11-01T04:17:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.238841+0000", 2023-11-01T04:17:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001311682 2023-11-01T04:17:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:21.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.527800+0000", 2023-11-01T04:17:21.436 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28895858200000002 2023-11-01T04:17:21.436 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.436 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.436 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.436 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:16:50.528051+0000", 2023-11-01T04:17:21.437 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000251325 2023-11-01T04:17:21.437 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.437 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.437 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.437 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.438 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:771 3.3s0 3:c78c701d:::smithi19039307-10:head [read 0~1753088,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88)", 2023-11-01T04:17:21.438 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:08.964104+0000", 2023-11-01T04:17:21.438 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 12.281597005, 2023-11-01T04:17:21.438 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61548556799999998, 2023-11-01T04:17:21.438 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.439 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:21.439 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.439 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:21.439 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:21.439 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 771 2023-11-01T04:17:21.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.440 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.440 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.440 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.440 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:08.964104+0000", 2023-11-01T04:17:21.441 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.441 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.441 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.441 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:08.964104+0000", 2023-11-01T04:17:21.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.442 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:08.964104+0000", 2023-11-01T04:17:21.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4600000000000004e-07 2023-11-01T04:17:21.443 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.443 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.443 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.443 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:08.964108+0000", 2023-11-01T04:17:21.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8229999999999997e-06 2023-11-01T04:17:21.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.444 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.444 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.444 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:08.964118+0000", 2023-11-01T04:17:21.444 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5000000000000005e-06 2023-11-01T04:17:21.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.445 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.445 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.445 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:08.964126+0000", 2023-11-01T04:17:21.445 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8499999999999994e-06 2023-11-01T04:17:21.445 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.446 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.446 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.446 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:08.967584+0000", 2023-11-01T04:17:21.446 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003457903 2023-11-01T04:17:21.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.447 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:21.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:08.967612+0000", 2023-11-01T04:17:21.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7990000000000001e-05 2023-11-01T04:17:21.447 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.447 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.448 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:09.067898+0000", 2023-11-01T04:17:21.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100286767 2023-11-01T04:17:21.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:09.068136+0000", 2023-11-01T04:17:21.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023774899999999999 2023-11-01T04:17:21.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:09.579589+0000", 2023-11-01T04:17:21.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51145333999999998 2023-11-01T04:17:21.450 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.450 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.450 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.451 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:781 3.2s0 3:4174a5c9:::smithi19039307-16:head [append 0~442368] snapc 35=[35,34,2f,28,19] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2023-11-01T04:17:21.451 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:10.060219+0000", 2023-11-01T04:17:21.451 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 11.185482025000001, 2023-11-01T04:17:21.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61637404600000001, 2023-11-01T04:17:21.452 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.452 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:21.452 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.452 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:21.453 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:21.453 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 781 2023-11-01T04:17:21.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.453 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.060219+0000", 2023-11-01T04:17:21.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.060219+0000", 2023-11-01T04:17:21.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.060222+0000", 2023-11-01T04:17:21.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.96e-06 2023-11-01T04:17:21.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.060817+0000", 2023-11-01T04:17:21.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00059549799999999997 2023-11-01T04:17:21.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.060853+0000", 2023-11-01T04:17:21.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5281999999999999e-05 2023-11-01T04:17:21.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.458 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.458 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.161035+0000", 2023-11-01T04:17:21.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100182085 2023-11-01T04:17:21.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.161210+0000", 2023-11-01T04:17:21.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017579499999999999 2023-11-01T04:17:21.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.161602+0000", 2023-11-01T04:17:21.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000391581 2023-11-01T04:17:21.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_started", 2023-11-01T04:17:21.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.164436+0000", 2023-11-01T04:17:21.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028338360000000002 2023-11-01T04:17:21.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_committed", 2023-11-01T04:17:21.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.177443+0000", 2023-11-01T04:17:21.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013007579999999999 2023-11-01T04:17:21.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2023-11-01T04:17:21.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.675825+0000", 2023-11-01T04:17:21.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.49838138900000001 2023-11-01T04:17:21.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.463 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.463 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.676593+0000", 2023-11-01T04:17:21.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00076804 2023-11-01T04:17:21.464 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.464 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.464 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.464 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:786 3.2s0 3:4174a5c9:::smithi19039307-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e89)", 2023-11-01T04:17:21.465 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:10.162966+0000", 2023-11-01T04:17:21.465 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 11.082734774, 2023-11-01T04:17:21.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82363005899999997, 2023-11-01T04:17:21.465 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.465 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:21.465 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.466 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:21.466 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:21.466 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 786 2023-11-01T04:17:21.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.466 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.162966+0000", 2023-11-01T04:17:21.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.162966+0000", 2023-11-01T04:17:21.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.162967+0000", 2023-11-01T04:17:21.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4130000000000001e-06 2023-11-01T04:17:21.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.162973+0000", 2023-11-01T04:17:21.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8660000000000002e-06 2023-11-01T04:17:21.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.162993+0000", 2023-11-01T04:17:21.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9332999999999999e-05 2023-11-01T04:17:21.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.163012+0000", 2023-11-01T04:17:21.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8839e-05 2023-11-01T04:17:21.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.179992+0000", 2023-11-01T04:17:21.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.016980367999999999 2023-11-01T04:17:21.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:21.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.180288+0000", 2023-11-01T04:17:21.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029603000000000002 2023-11-01T04:17:21.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.683028+0000", 2023-11-01T04:17:21.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.50274005499999996 2023-11-01T04:17:21.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.683242+0000", 2023-11-01T04:17:21.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000214236 2023-11-01T04:17:21.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:10.986596+0000", 2023-11-01T04:17:21.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30335391900000003 2023-11-01T04:17:21.476 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.476 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.476 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.477 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:941 3.2s0 3:42ba522a:::smithi19039307-20:head [read 0~3440640,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97)", 2023-11-01T04:17:21.477 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:19.169600+0000", 2023-11-01T04:17:21.477 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.0761004320000001, 2023-11-01T04:17:21.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70745689300000003, 2023-11-01T04:17:21.478 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.478 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:21.478 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.478 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:21.478 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:21.478 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 941 2023-11-01T04:17:21.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.479 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.169600+0000", 2023-11-01T04:17:21.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.169600+0000", 2023-11-01T04:17:21.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.169603+0000", 2023-11-01T04:17:21.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8930000000000001e-06 2023-11-01T04:17:21.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.169617+0000", 2023-11-01T04:17:21.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4059999999999999e-05 2023-11-01T04:17:21.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.169653+0000", 2023-11-01T04:17:21.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5746e-05 2023-11-01T04:17:21.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.269800+0000", 2023-11-01T04:17:21.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10014659099999999 2023-11-01T04:17:21.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.269960+0000", 2023-11-01T04:17:21.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016007900000000001 2023-11-01T04:17:21.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.270163+0000", 2023-11-01T04:17:21.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020292800000000001 2023-11-01T04:17:21.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.877057+0000", 2023-11-01T04:17:21.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60689459599999995 2023-11-01T04:17:21.487 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.487 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.487 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.488 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:942 3.2s0 3:42ba522a:::smithi19039307-20:head [read 0~3440640,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97)", 2023-11-01T04:17:21.488 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:19.269840+0000", 2023-11-01T04:17:21.488 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.9758612530000002, 2023-11-01T04:17:21.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.711364365, 2023-11-01T04:17:21.489 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.489 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:21.489 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.489 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:21.489 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:21.490 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 942 2023-11-01T04:17:21.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.490 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.269840+0000", 2023-11-01T04:17:21.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.269840+0000", 2023-11-01T04:17:21.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.492 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.269842+0000", 2023-11-01T04:17:21.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2500000000000001e-06 2023-11-01T04:17:21.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.269855+0000", 2023-11-01T04:17:21.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2830000000000001e-05 2023-11-01T04:17:21.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.269881+0000", 2023-11-01T04:17:21.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5888999999999999e-05 2023-11-01T04:17:21.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.369963+0000", 2023-11-01T04:17:21.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100082845 2023-11-01T04:17:21.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.370037+0000", 2023-11-01T04:17:21.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3854000000000001e-05 2023-11-01T04:17:21.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.370144+0000", 2023-11-01T04:17:21.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106325 2023-11-01T04:17:21.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.981204+0000", 2023-11-01T04:17:21.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61106037199999996 2023-11-01T04:17:21.498 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.498 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.499 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.499 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.4559.0:943 3.2s0 3:42ba522a:::smithi19039307-20:head [read 0~3440640,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97)", 2023-11-01T04:17:21.499 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2023-11-01T04:17:19.369990+0000", 2023-11-01T04:17:21.500 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.8757111499999999, 2023-11-01T04:17:21.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61494674199999999, 2023-11-01T04:17:21.500 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2023-11-01T04:17:21.500 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2023-11-01T04:17:21.500 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2023-11-01T04:17:21.500 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.4559", 2023-11-01T04:17:21.501 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:21.501 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 943 2023-11-01T04:17:21.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.501 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2023-11-01T04:17:21.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2023-11-01T04:17:21.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.369990+0000", 2023-11-01T04:17:21.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2023-11-01T04:17:21.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.369990+0000", 2023-11-01T04:17:21.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2023-11-01T04:17:21.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2023-11-01T04:17:21.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.369991+0000", 2023-11-01T04:17:21.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.316e-06 2023-11-01T04:17:21.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2023-11-01T04:17:21.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.369997+0000", 2023-11-01T04:17:21.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9669999999999998e-06 2023-11-01T04:17:21.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2023-11-01T04:17:21.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.370008+0000", 2023-11-01T04:17:21.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0558e-05 2023-11-01T04:17:21.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2023-11-01T04:17:21.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.370015+0000", 2023-11-01T04:17:21.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4170000000000003e-06 2023-11-01T04:17:21.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2023-11-01T04:17:21.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.370512+0000", 2023-11-01T04:17:21.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00049659699999999999 2023-11-01T04:17:21.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2023-11-01T04:17:21.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.370893+0000", 2023-11-01T04:17:21.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038184999999999998 2023-11-01T04:17:21.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2023-11-01T04:17:21.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2023-11-01T04:17:21.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2023-11-01T04:17:21.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2023-11-01T04:17:19.984936+0000", 2023-11-01T04:17:21.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61404303699999996 2023-11-01T04:17:21.510 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.510 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.510 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.510 INFO:teuthology.orchestra.run.smithi173.stdout: } 2023-11-01T04:17:21.510 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2023-11-01T04:17:21.511 INFO:teuthology.orchestra.run.smithi173.stdout:} 2023-11-01T04:17:21.511 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-11-01T04:17:21.513 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:17:21.510+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:17:21.513 INFO:tasks.rados.rados.0.smithi190.stdout:387: finishing copy_from racing read to smithi19039307-43 2023-11-01T04:17:21.513 INFO:tasks.rados.rados.0.smithi190.stdout:385: finishing rollback tid 0 to smithi19039307-18 2023-11-01T04:17:21.514 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-11-01T04:17:21.519 INFO:tasks.rados.rados.0.smithi190.stdout:385: finishing rollback tid 1 to smithi19039307-18 2023-11-01T04:17:21.519 INFO:tasks.rados.rados.0.smithi190.stdout:385: finishing rollback tid 2 to smithi19039307-18 2023-11-01T04:17:21.519 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 18 v 146 (ObjNum 98 snap 21 seq_num 98) dirty exists 2023-11-01T04:17:21.532 INFO:tasks.rados.rados.0.smithi190.stdout:388: finishing write tid 1 to smithi19039307-19 2023-11-01T04:17:21.573 INFO:tasks.rados.rados.0.smithi190.stdout:387: finishing copy_from to smithi19039307-43 2023-11-01T04:17:21.573 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 43 v 54 (ObjNum 66 snap 6 seq_num 66) dirty exists 2023-11-01T04:17:21.613 INFO:tasks.rados.rados.0.smithi190.stdout:384: finishing rollback tid 0 to smithi19039307-41 2023-11-01T04:17:21.613 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:21.608+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:21.649 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-11-01T04:17:21.649 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2023-11-01T04:17:21.650 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2023-11-01T04:17:21.650 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-11-01T04:17:21.663 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-11-01T04:17:21.664 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:17:21.664 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-11-01T04:17:21.684 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:17:21.684 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-11-01T04:17:21.703 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:17:21.703 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-11-01T04:17:21.708 INFO:tasks.rados.rados.0.smithi190.stdout:389: finishing write tid 1 to smithi19039307-13 2023-11-01T04:17:21.709 INFO:tasks.rados.rados.0.smithi190.stdout:389: finishing write tid 2 to smithi19039307-13 2023-11-01T04:17:21.710 INFO:tasks.rados.rados.0.smithi190.stdout:389: finishing write tid 3 to smithi19039307-13 2023-11-01T04:17:21.712 INFO:tasks.rados.rados.0.smithi190.stdout:388: finishing write tid 2 to smithi19039307-19 2023-11-01T04:17:21.713 INFO:tasks.rados.rados.0.smithi190.stdout:388: finishing write tid 3 to smithi19039307-19 2023-11-01T04:17:21.714 INFO:tasks.rados.rados.0.smithi190.stdout:384: finishing rollback tid 1 to smithi19039307-41 2023-11-01T04:17:21.714 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:21.710+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:21.715 INFO:tasks.rados.rados.0.smithi190.stdout:384: finishing rollback tid 2 to smithi19039307-41 2023-11-01T04:17:21.715 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 41 v 87 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-11-01T04:17:21.726 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:17:21.726 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-11-01T04:17:21.744 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:17:21.744 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-11-01T04:17:21.765 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:17:21.765 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-11-01T04:17:21.773 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 26 v 62 (ObjNum 25 snap 0 seq_num 25) dirty exists 2023-11-01T04:17:21.781 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:21.781 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:21.786 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:17:21.787 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-11-01T04:17:21.787 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-11-01T04:17:21.796 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-11-01T04:17:21.806 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:17:21.807 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'false' 2023-11-01T04:17:21.811 INFO:tasks.rados.rados.0.smithi190.stdout:389: finishing write tid 4 to smithi19039307-13 2023-11-01T04:17:21.812 INFO:tasks.rados.rados.0.smithi190.stdout:388: finishing write tid 4 to smithi19039307-19 2023-11-01T04:17:21.813 INFO:tasks.rados.rados.0.smithi190.stdout:388: finishing write tid 5 to smithi19039307-19 2023-11-01T04:17:21.814 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:21.812+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:21.829 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:17:21.830 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'false' 2023-11-01T04:17:21.895 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:17:21.895 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'false' 2023-11-01T04:17:21.916 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:21.914+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:21.921 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:17:21.922 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'false' 2023-11-01T04:17:21.922 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.12 2023-11-01T04:17:21.933 INFO:tasks.rados.rados.0.smithi190.stdout:389: finishing write tid 5 to smithi19039307-13 2023-11-01T04:17:21.949 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:17:21.949 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'false' 2023-11-01T04:17:21.965 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:21.965 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:21.974 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:17:21.975 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'false' 2023-11-01T04:17:21.980 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-11-01T04:17:22.000 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:17:22.000 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'false' 2023-11-01T04:17:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-01T04:17:22.005 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-01T04:17:22.017 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:17:22.016+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:17:22.092 INFO:teuthology.orchestra.run.smithi153.stderr:nodeep-scrub is set 2023-11-01T04:17:22.099 INFO:tasks.rados.rados.0.smithi190.stdout:384: done (6 left) 2023-11-01T04:17:22.099 INFO:tasks.rados.rados.0.smithi190.stdout:385: done (5 left) 2023-11-01T04:17:22.099 INFO:tasks.rados.rados.0.smithi190.stdout:387: done (4 left) 2023-11-01T04:17:22.099 INFO:tasks.rados.rados.0.smithi190.stdout:390: done (3 left) 2023-11-01T04:17:22.099 INFO:tasks.rados.rados.0.smithi190.stdout:391: done (2 left) 2023-11-01T04:17:22.100 INFO:tasks.rados.rados.0.smithi190.stdout:392: copy_from oid 38 from oid 37 current snap is 37 2023-11-01T04:17:22.100 INFO:tasks.rados.rados.0.smithi190.stdout:393: snap_remove snap 36 2023-11-01T04:17:22.117 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:22.114+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:22.149 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:22.150 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:22.164 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-11-01T04:17:22.256 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:22.254+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:22.319 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:17:22.316+0000 7f0bfc85b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:17:22.330 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:22.330 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:22.344 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-11-01T04:17:22.419 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:17:22.415+0000 7fd30c83e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:17:22.515 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:22.516 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:22.520 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:17:22.520+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:17:22.531 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2023-11-01T04:17:22.621 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:17:22.618+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:17:22.650 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:22.651 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:22.664 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2023-11-01T04:17:22.722 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:17:22.720+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:17:22.823 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:17:22.820+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:17:22.824 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:22.825 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:22.839 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2023-11-01T04:17:22.924 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:22.920+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:22.963 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:22.964 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:22.979 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2023-11-01T04:17:23.024 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:17:23.024+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:17:23.102 INFO:tasks.rados.rados.0.smithi190.stdout:393: done (3 left) 2023-11-01T04:17:23.103 INFO:tasks.rados.rados.0.smithi190.stdout:394: rollback oid 8 current snap is 37 2023-11-01T04:17:23.103 INFO:tasks.rados.rados.0.smithi190.stdout:rollback oid 8 to 28 2023-11-01T04:17:23.103 INFO:tasks.rados.rados.0.smithi190.stdout:388: finishing write tid 6 to smithi19039307-19 2023-11-01T04:17:23.103 INFO:tasks.rados.rados.0.smithi190.stdout:388: oid 19 updating version 0 to 40 2023-11-01T04:17:23.104 INFO:tasks.rados.rados.0.smithi190.stdout:388: oid 19 updating version 40 to 42 2023-11-01T04:17:23.104 INFO:tasks.rados.rados.0.smithi190.stdout:388: oid 19 version 42 is already newer than 39 2023-11-01T04:17:23.104 INFO:tasks.rados.rados.0.smithi190.stdout:388: oid 19 version 42 is already newer than 41 2023-11-01T04:17:23.104 INFO:tasks.rados.rados.0.smithi190.stdout:388: oid 19 updating version 42 to 43 2023-11-01T04:17:23.104 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 19 v 43 (ObjNum 139 snap 36 seq_num 139) dirty exists 2023-11-01T04:17:23.105 INFO:tasks.rados.rados.0.smithi190.stdout:388: left oid 19 (ObjNum 139 snap 36 seq_num 139) 2023-11-01T04:17:23.105 INFO:tasks.rados.rados.0.smithi190.stdout:389: finishing write tid 6 to smithi19039307-13 2023-11-01T04:17:23.105 INFO:tasks.rados.rados.0.smithi190.stdout:389: oid 13 updating version 0 to 167 2023-11-01T04:17:23.105 INFO:tasks.rados.rados.0.smithi190.stdout:389: oid 13 version 167 is already newer than 165 2023-11-01T04:17:23.105 INFO:tasks.rados.rados.0.smithi190.stdout:389: oid 13 version 167 is already newer than 163 2023-11-01T04:17:23.106 INFO:tasks.rados.rados.0.smithi190.stdout:389: oid 13 version 167 is already newer than 166 2023-11-01T04:17:23.106 INFO:tasks.rados.rados.0.smithi190.stdout:389: oid 13 version 167 is already newer than 164 2023-11-01T04:17:23.106 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 13 v 167 (ObjNum 140 snap 36 seq_num 140) dirty exists 2023-11-01T04:17:23.106 INFO:tasks.rados.rados.0.smithi190.stdout:389: left oid 13 (ObjNum 140 snap 36 seq_num 140) 2023-11-01T04:17:23.106 INFO:tasks.rados.rados.0.smithi190.stdout:392: finishing copy_from racing read to smithi19039307-38 2023-11-01T04:17:23.107 INFO:tasks.rados.rados.0.smithi190.stdout:388: done (3 left) 2023-11-01T04:17:23.107 INFO:tasks.rados.rados.0.smithi190.stdout:389: done (2 left) 2023-11-01T04:17:23.107 INFO:tasks.rados.rados.0.smithi190.stdout:395: read oid 20 snap -1 2023-11-01T04:17:23.107 INFO:tasks.rados.rados.0.smithi190.stdout:395: expect (ObjNum 110 snap 24 seq_num 110) 2023-11-01T04:17:23.107 INFO:tasks.rados.rados.0.smithi190.stdout:396: read oid 36 snap -1 2023-11-01T04:17:23.108 INFO:tasks.rados.rados.0.smithi190.stdout:396: expect (ObjNum 53 snap 1 seq_num 53) 2023-11-01T04:17:23.108 INFO:tasks.rados.rados.0.smithi190.stdout:397: delete oid 17 current snap is 37 2023-11-01T04:17:23.108 INFO:tasks.rados.rados.0.smithi190.stdout:392: finishing copy_from to smithi19039307-38 2023-11-01T04:17:23.108 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 38 v 60 (ObjNum 87 snap 20 seq_num 87) dirty exists 2023-11-01T04:17:23.114 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:23.114 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:23.125 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:17:23.124+0000 7fb28fa0f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:17:23.128 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2023-11-01T04:17:23.215 INFO:tasks.rados.rados.0.smithi190.stdout:392: done (4 left) 2023-11-01T04:17:23.215 INFO:tasks.rados.rados.0.smithi190.stdout:397: done (3 left) 2023-11-01T04:17:23.216 INFO:tasks.rados.rados.0.smithi190.stdout:append oid 13 current snap is 37 2023-11-01T04:17:23.216 INFO:tasks.rados.rados.0.smithi190.stdout:398: seq_num 142 ranges {2473984=442368,2916352=720896,3637248=540672,4177920=688128} 2023-11-01T04:17:23.226 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:17:23.222+0000 7f4052983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:17:23.251 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:23.252 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:23.259 INFO:tasks.rados.rados.0.smithi190.stdout:398: writing smithi19039307-13 from 2473984 to 2916352 tid 1 2023-11-01T04:17:23.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2023-11-01T04:17:23.269 INFO:tasks.rados.rados.0.smithi190.stdout:398: writing smithi19039307-13 from 2916352 to 3637248 tid 2 2023-11-01T04:17:23.275 INFO:tasks.rados.rados.0.smithi190.stdout:398: writing smithi19039307-13 from 3637248 to 4177920 tid 3 2023-11-01T04:17:23.283 INFO:tasks.rados.rados.0.smithi190.stdout:398: writing smithi19039307-13 from 4177920 to 4866048 tid 4 2023-11-01T04:17:23.283 INFO:tasks.rados.rados.0.smithi190.stdout:399: delete oid 9 current snap is 37 2023-11-01T04:17:23.328 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:17:23.324+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:17:23.408 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2023-11-01T04:17:23.427 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:23.432 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:23.432 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:23.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-11-01T04:17:23.489 INFO:tasks.rados.rados.0.smithi190.stdout:399: done (4 left) 2023-11-01T04:17:23.489 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 4 2023-11-01T04:17:23.526 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:23.527 INFO:tasks.rados.rados.0.smithi190.stdout:394: finishing rollback tid 1 to smithi19039307-8 2023-11-01T04:17:23.529 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:23.530 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['config', 'set', 'filestore_inject_stall', '3'] 2023-11-01T04:17:23.531 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:23.526+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:23.578 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:23.578 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:23.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-11-01T04:17:23.607 INFO:tasks.rados.rados.0.smithi190.stdout:394: finishing rollback tid 2 to smithi19039307-8 2023-11-01T04:17:23.607 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 8 v 116 (ObjNum 108 snap 24 seq_num 108) dirty exists 2023-11-01T04:17:23.607 INFO:tasks.rados.rados.0.smithi190.stdout:394: done (3 left) 2023-11-01T04:17:23.607 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 3 2023-11-01T04:17:23.652 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:17:23.652+0000 7f5c97276640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:17:23.724 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:23.724 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:23.727 INFO:tasks.rados.rados.0.smithi190.stdout:398: finishing write tid 1 to smithi19039307-13 2023-11-01T04:17:23.728 INFO:tasks.rados.rados.0.smithi190.stdout:398: finishing write tid 2 to smithi19039307-13 2023-11-01T04:17:23.728 INFO:tasks.rados.rados.0.smithi190.stdout:396: expect (ObjNum 53 snap 1 seq_num 53) 2023-11-01T04:17:23.731 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:17:23.728+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:17:23.739 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-11-01T04:17:23.829 INFO:tasks.rados.rados.0.smithi190.stdout:398: finishing write tid 3 to smithi19039307-13 2023-11-01T04:17:23.829 INFO:tasks.rados.rados.0.smithi190.stdout:395: expect (ObjNum 110 snap 24 seq_num 110) 2023-11-01T04:17:23.832 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:17:23.830+0000 7f63a4983640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:17:23.911 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2023-11-01T04:17:23.911 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-11-01T04:17:23.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-11-01T04:17:23.932 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2023-11-01T04:17:23.953 INFO:tasks.rados.rados.0.smithi190.stdout:398: finishing write tid 4 to smithi19039307-13 2023-11-01T04:17:23.953 INFO:tasks.rados.rados.0.smithi190.stdout:398: finishing write tid 5 to smithi19039307-13 2023-11-01T04:17:23.954 INFO:tasks.rados.rados.0.smithi190.stdout:395: done (2 left) 2023-11-01T04:17:23.954 INFO:tasks.rados.rados.0.smithi190.stdout:396: done (1 left) 2023-11-01T04:17:23.954 INFO:tasks.rados.rados.0.smithi190.stdout: waiting on 1 2023-11-01T04:17:24.006 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-11-01T04:17:24.030 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:24.033 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:24.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-11-01T04:17:24.035 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:24.032+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:24.037 INFO:tasks.rados.rados.0.smithi190.stdout:398: finishing write tid 6 to smithi19039307-13 2023-11-01T04:17:24.037 INFO:tasks.rados.rados.0.smithi190.stdout:398: oid 13 updating version 0 to 170 2023-11-01T04:17:24.037 INFO:tasks.rados.rados.0.smithi190.stdout:398: oid 13 updating version 170 to 172 2023-11-01T04:17:24.037 INFO:tasks.rados.rados.0.smithi190.stdout:398: oid 13 version 172 is already newer than 169 2023-11-01T04:17:24.037 INFO:tasks.rados.rados.0.smithi190.stdout:398: oid 13 version 172 is already newer than 171 2023-11-01T04:17:24.038 INFO:tasks.rados.rados.0.smithi190.stdout:398: oid 13 version 172 is already newer than 168 2023-11-01T04:17:24.038 INFO:tasks.rados.rados.0.smithi190.stdout:update_object_version oid 13 v 172 (ObjNum 142 snap 37 seq_num 142) dirty exists 2023-11-01T04:17:24.038 INFO:tasks.rados.rados.0.smithi190.stdout:398: left oid 13 (ObjNum 142 snap 37 seq_num 142) 2023-11-01T04:17:24.038 INFO:tasks.rados.rados.0.smithi190.stdout:398: done (0 left) 2023-11-01T04:17:24.045 INFO:tasks.rados.rados.0.smithi190.stderr:0 errors. 2023-11-01T04:17:24.045 INFO:tasks.rados.rados.0.smithi190.stderr: 2023-11-01T04:17:24.056 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-01T04:17:24.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-01T04:17:24.139 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:24.142 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:24.143 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-11-01T04:17:24.160 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:17:24.159+0000 7fa483489640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:17:24.166 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0: {} 2023-11-01T04:17:24.166 INFO:teuthology.orchestra.run.smithi153.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-11-01T04:17:24.188 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1: {} 2023-11-01T04:17:24.188 INFO:teuthology.orchestra.run.smithi153.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-11-01T04:17:24.209 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2: {} 2023-11-01T04:17:24.209 INFO:teuthology.orchestra.run.smithi153.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-11-01T04:17:24.237 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3: {} 2023-11-01T04:17:24.237 INFO:teuthology.orchestra.run.smithi153.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-11-01T04:17:24.238 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:17:24.236+0000 7f04e4e7a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:17:24.255 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4: {} 2023-11-01T04:17:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-11-01T04:17:24.275 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5: {} 2023-11-01T04:17:24.275 INFO:teuthology.orchestra.run.smithi153.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-11-01T04:17:24.293 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:24.295 INFO:teuthology.orchestra.run.smithi153.stdout:osd.6: {} 2023-11-01T04:17:24.295 INFO:teuthology.orchestra.run.smithi153.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-11-01T04:17:24.296 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.7 2023-11-01T04:17:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:24.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2023-11-01T04:17:24.312 INFO:teuthology.orchestra.run.smithi153.stdout:osd.8: {} 2023-11-01T04:17:24.312 INFO:teuthology.orchestra.run.smithi153.stderr:osd.8: osd_enable_op_tracker = 'true' 2023-11-01T04:17:24.339 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:24.336+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:24.345 INFO:teuthology.orchestra.run.smithi153.stdout:osd.9: {} 2023-11-01T04:17:24.346 INFO:teuthology.orchestra.run.smithi153.stderr:osd.9: osd_enable_op_tracker = 'true' 2023-11-01T04:17:24.354 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:17:24.355 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":101,"fsid":"0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d","created":"2023-11-01T04:15:44.902210+0000","modified":"2023-11-01T04:17:23.092469+0000","last_up_change":"2023-11-01T04:16:58.831691+0000","last_in_change":"2023-11-01T04:16:52.779010+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-01T04:15:57.685107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-01T04:15:59.882949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2023-11-01T04:16:08.908876+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":65,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"83ce3480-defe-4c73-aa1d-47c7941a23ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6817","nonce":59702962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6819","nonce":59702962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6823","nonce":59702962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6821","nonce":59702962}]},"public_addr":"172.21.15.153:6817/59702962","cluster_addr":"172.21.15.153:6819/59702962","heartbeat_back_addr":"172.21.15.153:6823/59702962","heartbeat_front_addr":"172.21.15.153:6821/59702962","state":["exists","up"]},{"osd":1,"uuid":"01a9801f-72fb-4299-8708-d70eb59b0fb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6815","nonce":3103315277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6817","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6818","nonce":3103315277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6823","nonce":3103315277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6821","nonce":3103315277}]},"public_addr":"172.21.15.173:6815/3103315277","cluster_addr":"172.21.15.173:6818/3103315277","heartbeat_back_addr":"172.21.15.173:6823/3103315277","heartbeat_front_addr":"172.21.15.173:6821/3103315277","state":["exists","up"]},{"osd":2,"uuid":"7c16db19-9bec-4ca0-88eb-bb5fb673d160","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6817","nonce":1072240131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6820","nonce":1072240131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6825","nonce":1072240131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6821","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6823","nonce":1072240131}]},"public_addr":"172.21.15.181:6817/1072240131","cluster_addr":"172.21.15.181:6820/1072240131","heartbeat_back_addr":"172.21.15.181:6825/1072240131","heartbeat_front_addr":"172.21.15.181:6823/1072240131","state":["exists","up"]},{"osd":3,"uuid":"4c4cc8d1-6761-4d20-90e8-7c3d43d889d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6827","nonce":2304315422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6829","nonce":2304315422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6833","nonce":2304315422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6831","nonce":2304315422}]},"public_addr":"172.21.15.190:6827/2304315422","cluster_addr":"172.21.15.190:6829/2304315422","heartbeat_back_addr":"172.21.15.190:6833/2304315422","heartbeat_front_addr":"172.21.15.190:6831/2304315422","state":["exists","up"]},{"osd":4,"uuid":"0fb14630-87e6-4456-9640-7986a5079821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6825","nonce":3209586931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6827","nonce":3209586931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6831","nonce":3209586931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6829","nonce":3209586931}]},"public_addr":"172.21.15.153:6825/3209586931","cluster_addr":"172.21.15.153:6827/3209586931","heartbeat_back_addr":"172.21.15.153:6831/3209586931","heartbeat_front_addr":"172.21.15.153:6829/3209586931","state":["exists","up"]},{"osd":5,"uuid":"cc715599-0461-477b-beb2-8777d6f36b22","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6825","nonce":409879231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6827","nonce":409879231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6831","nonce":409879231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6829","nonce":409879231}]},"public_addr":"172.21.15.173:6825/409879231","cluster_addr":"172.21.15.173:6827/409879231","heartbeat_back_addr":"172.21.15.173:6831/409879231","heartbeat_front_addr":"172.21.15.173:6829/409879231","state":["exists","up"]},{"osd":6,"uuid":"ca5bea1c-f309-4258-b85d-738097037962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6819","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6822","nonce":3465541770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6827","nonce":3465541770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6831","nonce":3465541770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6829","nonce":3465541770}]},"public_addr":"172.21.15.181:6822/3465541770","cluster_addr":"172.21.15.181:6827/3465541770","heartbeat_back_addr":"172.21.15.181:6831/3465541770","heartbeat_front_addr":"172.21.15.181:6829/3465541770","state":["exists","up"]},{"osd":7,"uuid":"1e80e965-2a46-468c-bcb8-49bee8dd19c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6803","nonce":1763196448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6805","nonce":1763196448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6809","nonce":1763196448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6807","nonce":1763196448}]},"public_addr":"172.21.15.190:6803/1763196448","cluster_addr":"172.21.15.190:6805/1763196448","heartbeat_back_addr":"172.21.15.190:6809/1763196448","heartbeat_front_addr":"172.21.15.190:6807/1763196448","state":["exists"]},{"osd":8,"uuid":"87179b74-818f-47df-ad70-1392cfe227e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6801","nonce":1068520389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6803","nonce":1068520389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6807","nonce":1068520389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6805","nonce":1068520389}]},"public_addr":"172.21.15.153:6801/1068520389","cluster_addr":"172.21.15.153:6803/1068520389","heartbeat_back_addr":"172.21.15.153:6807/1068520389","heartbeat_front_addr":"172.21.15.153:6805/1068520389","state":["exists","up"]},{"osd":9,"uuid":"10d3c443-1748-4a7c-8b60-b58cfd565b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6801","nonce":210565898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6803","nonce":210565898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6807","nonce":210565898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6805","nonce":210565898}]},"public_addr":"172.21.15.173:6801/210565898","cluster_addr":"172.21.15.173:6803/210565898","heartbeat_back_addr":"172.21.15.173:6807/210565898","heartbeat_front_addr":"172.21.15.173:6805/210565898","state":["exists","up"]},{"osd":10,"uuid":"81bb6569-d02b-4638-8b76-f99756d415b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6801","nonce":2007605188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6803","nonce":2007605188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6807","nonce":2007605188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6805","nonce":2007605188}]},"public_addr":"172.21.15.181:6801/2007605188","cluster_addr":"172.21.15.181:6803/2007605188","heartbeat_back_addr":"172.21.15.181:6807/2007605188","heartbeat_front_addr":"172.21.15.181:6805/2007605188","state":["exists","up"]},{"osd":11,"uuid":"b8aefe11-845f-48a4-930c-b05cdfbb460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6812","nonce":2456507663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6816","nonce":2456507663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6824","nonce":2456507663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6820","nonce":2456507663}]},"public_addr":"172.21.15.190:6812/2456507663","cluster_addr":"172.21.15.190:6816/2456507663","heartbeat_back_addr":"172.21.15.190:6824/2456507663","heartbeat_front_addr":"172.21.15.190:6820/2456507663","state":["exists","up"]},{"osd":12,"uuid":"2165437d-26ba-4e50-8646-8cedbc10da53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":68,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6809","nonce":819245857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6811","nonce":819245857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6815","nonce":819245857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6813","nonce":819245857}]},"public_addr":"172.21.15.153:6809/819245857","cluster_addr":"172.21.15.153:6811/819245857","heartbeat_back_addr":"172.21.15.153:6815/819245857","heartbeat_front_addr":"172.21.15.153:6813/819245857","state":["exists"]},{"osd":13,"uuid":"7b9f38b4-f5ff-43f1-a373-2aaed710026d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6809","nonce":3482699195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6811","nonce":3482699195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6819","nonce":3482699195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6814","nonce":3482699195}]},"public_addr":"172.21.15.173:6809/3482699195","cluster_addr":"172.21.15.173:6811/3482699195","heartbeat_back_addr":"172.21.15.173:6819/3482699195","heartbeat_front_addr":"172.21.15.173:6814/3482699195","state":["exists","up"]},{"osd":14,"uuid":"054b27d2-8a4d-4851-8af8-78b71e6a537e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6809","nonce":2780345165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6811","nonce":2780345165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6815","nonce":2780345165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6813","nonce":2780345165}]},"public_addr":"172.21.15.181:6809/2780345165","cluster_addr":"172.21.15.181:6811/2780345165","heartbeat_back_addr":"172.21.15.181:6815/2780345165","heartbeat_front_addr":"172.21.15.181:6813/2780345165","state":["exists","up"]},{"osd":15,"uuid":"e650f936-3712-47cd-9b17-d732991a27a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6813","nonce":3871145029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6815","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6817","nonce":3871145029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6823","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6825","nonce":3871145029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6819","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6821","nonce":3871145029}]},"public_addr":"172.21.15.190:6813/3871145029","cluster_addr":"172.21.15.190:6817/3871145029","heartbeat_back_addr":"172.21.15.190:6825/3871145029","heartbeat_front_addr":"172.21.15.190:6821/3871145029","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:08.233469+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:12.717062+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:03.140612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:03.367734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:20.311480+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":52947,"last_purged_snaps_scrub":"2023-11-01T04:17:20.365999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:18.902496+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.028107+0000","dead_epoch":78},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:10.612247+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:08.793386+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:16:59.158158+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:16:58.884095+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.650893+0000","dead_epoch":68},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:21.087831+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:13.468022+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:06.291345+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.4","osds":[9,8]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"3.e","osds":[11,9,1,4,2,3,5]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":4,"length":1},{"begin":11,"length":1},{"begin":17,"length":2},{"begin":21,"length":1},{"begin":24,"length":1},{"begin":27,"length":2},{"begin":30,"length":1},{"begin":34,"length":1},{"begin":38,"length":2},{"begin":47,"length":2},{"begin":51,"length":1},{"begin":56,"length":1},{"begin":64,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":64,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-01T04:17:24.372 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-01T04:17:24.379 INFO:teuthology.orchestra.run.smithi153.stdout:osd.10: {} 2023-11-01T04:17:24.380 INFO:teuthology.orchestra.run.smithi153.stderr:osd.10: osd_enable_op_tracker = 'true' 2023-11-01T04:17:24.408 INFO:teuthology.orchestra.run.smithi153.stdout:osd.11: {} 2023-11-01T04:17:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:osd.11: osd_enable_op_tracker = 'true' 2023-11-01T04:17:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:Error ENXIO: problem getting command descriptions from osd.12 2023-11-01T04:17:24.440 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:17:24.438+0000 7fc4f431f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:17:24.440 INFO:teuthology.orchestra.run.smithi153.stdout:osd.13: {} 2023-11-01T04:17:24.441 INFO:teuthology.orchestra.run.smithi153.stderr:osd.13: osd_enable_op_tracker = 'true' 2023-11-01T04:17:24.460 INFO:teuthology.orchestra.run.smithi153.stdout:osd.14: {} 2023-11-01T04:17:24.460 INFO:teuthology.orchestra.run.smithi153.stderr:osd.14: osd_enable_op_tracker = 'true' 2023-11-01T04:17:24.461 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:17:24.461 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:17:24.462 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2023-11-01T04:17:24.462 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:17:24.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2023-11-01T04:17:24.477 INFO:teuthology.orchestra.run.smithi153.stdout:osd.15: {} 2023-11-01T04:17:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:osd.15: osd_enable_op_tracker = 'true' 2023-11-01T04:17:24.481 DEBUG:teuthology.orchestra.run:got remote process result: 6 2023-11-01T04:17:24.481 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2023-11-01T04:17:24.540 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:17:24.540+0000 7f9bdf92c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:17:24.597 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:17:24.597 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:17:24.598 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2023-11-01T04:17:24.598 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2023-11-01T04:17:24.598 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:17:24.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2023-11-01T04:17:24.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~30s 2023-11-01T04:17:24.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~22s 2023-11-01T04:17:24.641 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:17:24.638+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:17:24.669 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:17:24.670 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2023-11-01T04:17:24.688 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2023-11-01T04:17:23.632091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":150471200,"num_objects":164,"num_object_clones":112,"num_object_copies":1138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":132,"num_objects_misplaced":13,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":5,"num_read":622,"num_read_kb":209916,"num_write":973,"num_write_kb":159556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":29720576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1310,"ondisk_log_size":1310,"up":129,"acting":129,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":693964,"kb_used_data":255380,"kb_used_omap":23,"kb_used_meta":438376,"kb_avail":1414883636,"statfs":{"total":1449551462400,"available":1448840843264,"internally_reserved":0,"allocated":261509120,"data_stored":397135270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23911,"internal_metadata":448897689},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19202048,"num_objects":14,"num_object_clones":14,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":25,"num_read_kb":33635,"num_write":64,"num_write_kb":18752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001492"},"pg_stats":[{"pgid":"3.f","version":"81'26","reported_seq":284,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-11-01T04:17:22.252785+0000","last_change":"2023-11-01T04:17:18.766730+0000","last_active":"2023-11-01T04:17:22.252785+0000","last_peered":"2023-11-01T04:17:22.252785+0000","last_clean":"2023-11-01T04:17:22.252785+0000","last_became_active":"2023-11-01T04:16:11.171434+0000","last_became_peered":"2023-11-01T04:16:11.171434+0000","last_unstale":"2023-11-01T04:17:22.252785+0000","last_undegraded":"2023-11-01T04:17:22.252785+0000","last_fullsized":"2023-11-01T04:17:22.252785+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013562310000000001,"stat_sum":{"num_bytes":557056,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":3308,"num_write":34,"num_write_kb":544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,15,13,14,11,2],"acting":[0,3,15,13,14,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"2"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1b","length":"2"},{"start":"1e","length":"1"},{"start":"22","length":"1"},{"start":"26","length":"2"},{"start":"2f","length":"2"},{"start":"33","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.e","version":"99'167","reported_seq":434,"reported_epoch":100,"state":"active+clean+remapped","last_fresh":"2023-11-01T04:17:22.354301+0000","last_change":"2023-11-01T04:17:18.984128+0000","last_active":"2023-11-01T04:17:22.354301+0000","last_peered":"2023-11-01T04:17:22.354301+0000","last_clean":"2023-11-01T04:17:22.354301+0000","last_became_active":"2023-11-01T04:16:55.067056+0000","last_became_peered":"2023-11-01T04:16:55.067056+0000","last_unstale":"2023-11-01T04:17:22.354301+0000","last_undegraded":"2023-11-01T04:17:22.354301+0000","last_fullsized":"2023-11-01T04:17:22.354301+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31290905699999999,"stat_sum":{"num_bytes":13975552,"num_objects":13,"num_object_clones":8,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":45,"num_read_kb":19120,"num_write":52,"num_write_kb":16608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1,4,2,3,2147483647],"acting":[11,9,1,4,2,3,5],"avail_no_missing":["11(0)","1(2)","2(4)","3(5)","4(3)","5(6)","9(1)"],"object_location_counts":[{"shards":"1(2),2(4),3(5),4(3),5(6),9(1),11(0)","objects":13}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"4","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"2"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1b","length":"2"},{"start":"1e","length":"1"},{"start":"22","length":"1"},{"start":"26","length":"2"},{"start":"2f","length":"2"},{"start":"33","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.d","version":"97'114","reported_seq":341,"reported_epoch":100,"state":"active+undersized+degraded","last_fresh":"2023-11-01T04:17:22.108118+0000","last_change":"2023-11-01T04:16:49.188704+0000","last_active":"2023-11-01T04:17:22.108118+0000","last_peered":"2023-11-01T04:17:22.108118+0000","last_clean":"2023-11-01T04:16:47.392467+0000","last_became_active":"2023-11-01T04:16:49.188704+0000","last_became_peered":"2023-11-01T04:16:49.188704+0000","last_unstale":"2023-11-01T04:17:22.108118+0000","last_undegraded":"2023-11-01T04:16:48.885189+0000","last_fullsized":"2023-11-01T04:16:48.884591+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:30.445372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014181370000000001,"stat_sum":{"num_bytes":17580032,"num_objects":23,"num_object_clones":16,"num_object_copies":161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":62,"num_read_kb":21333,"num_write":114,"num_write_kb":18224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,4,11,0,8,2,6],"acting":[2147483647,4,11,0,8,2,6],"avail_no_missing":["4(1)","0(3)","2(5)","6(6)","8(4)","11(2)"],"object_location_counts":[{"shards":"0(3),2(5),4(1),6(6),8(4),11(2)","objects":23}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"79'4","reported_seq":217,"reported_epoch":100,"state":"active+undersized+degraded","last_fresh":"2023-11-01T04:17:22.252699+0000","last_change":"2023-11-01T04:17:00.181148+0000","last_active":"2023-11-01T04:17:22.252699+0000","last_peered":"2023-11-01T04:17:22.252699+0000","last_clean":"2023-11-01T04:16:46.880755+0000","last_became_active":"2023-11-01T04:17:00.181148+0000","last_became_peered":"2023-11-01T04:17:00.181148+0000","last_unstale":"2023-11-01T04:17:22.252699+0000","last_undegraded":"2023-11-01T04:16:59.871360+0000","last_fullsized":"2023-11-01T04:16:59.870955+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00153498,"stat_sum":{"num_bytes":212992,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":290,"num_write":3,"num_write_kb":208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,2147483647,3,2147483647,2,6],"acting":[0,10,2147483647,3,2147483647,2,6],"avail_no_missing":["0(0)","2(5)","3(3)","6(6)","10(1)"],"object_location_counts":[{"shards":"0(0),2(5),3(3),6(6),10(1)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"1"}]},{"pgid":"3.b","version":"99'62","reported_seq":310,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-11-01T04:17:22.252168+0000","last_change":"2023-11-01T04:17:18.983867+0000","last_active":"2023-11-01T04:17:22.252168+0000","last_peered":"2023-11-01T04:17:22.252168+0000","last_clean":"2023-11-01T04:17:22.252168+0000","last_became_active":"2023-11-01T04:16:11.228189+0000","last_became_peered":"2023-11-01T04:16:11.228189+0000","last_unstale":"2023-11-01T04:17:22.252168+0000","last_undegraded":"2023-11-01T04:17:22.252168+0000","last_fullsized":"2023-11-01T04:17:22.252168+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21193651399999999,"stat_sum":{"num_bytes":999424,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":56,"num_write_kb":3520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1,9,11,4,14],"acting":[10,6,1,9,11,4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"4","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"2"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1b","length":"2"},{"start":"1e","length":"1"},{"start":"22","length":"1"},{"start":"26","length":"2"},{"start":"2f","length":"2"},{"start":"33","length":"1"},{"start":"38","length":"1"}]},{"pgid":"3.a","version":"99'43","reported_seq":257,"reported_epoch":100,"state":"active+undersized+degraded","last_fresh":"2023-11-01T04:17:22.209245+0000","last_change":"2023-11-01T04:16:49.089977+0000","last_active":"2023-11-01T04:17:22.209245+0000","last_peered":"2023-11-01T04:17:22.209245+0000","last_clean":"2023-11-01T04:16:47.392776+0000","last_became_active":"2023-11-01T04:16:49.089977+0000","last_became_peered":"2023-11-01T04:16:49.089977+0000","last_unstale":"2023-11-01T04:17:22.209245+0000","last_undegraded":"2023-11-01T04:16:48.785049+0000","last_fullsized":"2023-11-01T04:16:48.784723+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:25.682138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001008197,"stat_sum":{"num_bytes":7569408,"num_objects":8,"num_object_clones":6,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":22,"num_read_kb":11481,"num_write":26,"num_write_kb":8160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,4,1,9,3,15,8],"acting":[2147483647,4,1,9,3,15,8],"avail_no_missing":["4(1)","1(2)","3(4)","8(6)","9(3)","15(5)"],"object_location_counts":[{"shards":"1(2),3(4),4(1),8(6),9(3),15(5)","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"99'111","reported_seq":336,"reported_epoch":100,"state":"active+undersized+degraded","last_fresh":"2023-11-01T04:17:22.253693+0000","last_change":"2023-11-01T04:17:00.080055+0000","last_active":"2023-11-01T04:17:22.253693+0000","last_peered":"2023-11-01T04:17:22.253693+0000","last_clean":"2023-11-01T04:16:57.459351+0000","last_became_active":"2023-11-01T04:17:00.080055+0000","last_became_peered":"2023-11-01T04:17:00.080055+0000","last_unstale":"2023-11-01T04:17:22.253693+0000","last_undegraded":"2023-11-01T04:16:59.871565+0000","last_fullsized":"2023-11-01T04:16:59.870499+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:33.282758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.617598066,"stat_sum":{"num_bytes":14811136,"num_objects":15,"num_object_clones":12,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":38,"num_read_kb":8561,"num_write":82,"num_write_kb":15840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,1,14,11,4,10,3],"acting":[2147483647,1,14,11,4,10,3],"avail_no_missing":["1(1)","3(6)","4(4)","10(5)","11(3)","14(2)"],"object_location_counts":[{"shards":"1(1),3(6),4(4),10(5),11(3),14(2)","objects":15}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"},{"start":"b","length":"1"},{"start":"15","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"1"}]},{"pgid":"3.8","version":"97'70","reported_seq":147,"reported_epoch":100,"state":"active+undersized+degraded","last_fresh":"2023-11-01T04:17:22.254565+0000","last_change":"2023-11-01T04:17:00.274968+0000","last_active":"2023-11-01T04:17:22.254565+0000","last_peered":"2023-11-01T04:17:22.254565+0000","last_clean":"2023-11-01T04:16:52.198227+0000","last_became_active":"2023-11-01T04:17:00.274968+0000","last_became_peered":"2023-11-01T04:17:00.274968+0000","last_unstale":"2023-11-01T04:17:22.254565+0000","last_undegraded":"2023-11-01T04:16:59.966334+0000","last_fullsized":"2023-11-01T04:16:59.965735+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:25.411512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.22409670200000001,"stat_sum":{"num_bytes":14385152,"num_objects":14,"num_object_clones":11,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":37,"num_read_kb":24590,"num_write":54,"num_write_kb":14048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,11,6,14,0,2147483647],"acting":[3,10,11,6,14,0,2147483647],"avail_no_missing":["3(0)","0(5)","6(3)","10(1)","11(2)","14(4)"],"object_location_counts":[{"shards":"0(5),3(0),6(3),10(1),11(2),14(4)","objects":14}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b","length":"1"},{"start":"15","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"1"}]},{"pgid":"1.0","version":"33'157","reported_seq":283,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-11-01T04:17:22.197433+0000","last_change":"2023-11-01T04:15:59.016143+0000","last_active":"2023-11-01T04:17:22.197433+0000","last_peered":"2023-11-01T04:17:22.197433+0000","last_clean":"2023-11-01T04:17:22.197433+0000","last_became_active":"2023-11-01T04:15:59.014137+0000","last_became_peered":"2023-11-01T04:15:59.014137+0000","last_unstale":"2023-11-01T04:17:22.197433+0000","last_undegraded":"2023-11-01T04:17:22.197433+0000","last_fullsized":"2023-11-01T04:17:22.197433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:15:57.686019+0000","last_clean_scrub_stamp":"2023-11-01T04:15:57.686019+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":45,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-11-01T04:17:21.099576+0000","last_change":"2023-11-01T04:17:14.160921+0000","last_active":"2023-11-01T04:17:21.099576+0000","last_peered":"2023-11-01T04:17:21.099576+0000","last_clean":"2023-11-01T04:17:21.099576+0000","last_became_active":"2023-11-01T04:16:54.109999+0000","last_became_peered":"2023-11-01T04:16:54.109999+0000","last_unstale":"2023-11-01T04:17:21.099576+0000","last_undegraded":"2023-11-01T04:17:21.099576+0000","last_fullsized":"2023-11-01T04:17:21.099576+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:17:14.160606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:17:14.160606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:18:16.720019+0000","scrub_duration":0.20386212000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13],"acting":[0,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"99'146","reported_seq":391,"reported_epoch":101,"state":"active+undersized+degraded","last_fresh":"2023-11-01T04:17:23.401948+0000","last_change":"2023-11-01T04:17:00.072763+0000","last_active":"2023-11-01T04:17:23.401948+0000","last_peered":"2023-11-01T04:17:23.401948+0000","last_clean":"2023-11-01T04:16:57.952047+0000","last_became_active":"2023-11-01T04:17:00.072763+0000","last_became_peered":"2023-11-01T04:17:00.072763+0000","last_unstale":"2023-11-01T04:17:23.401948+0000","last_undegraded":"2023-11-01T04:16:59.871806+0000","last_fullsized":"2023-11-01T04:16:59.870638+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:37.978422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001800704,"stat_sum":{"num_bytes":24657920,"num_objects":19,"num_object_clones":13,"num_object_copies":133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":52,"num_read_kb":31281,"num_write":72,"num_write_kb":25152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,3,0,4,8,1,2147483647],"acting":[13,3,0,4,8,1,2147483647],"avail_no_missing":["13(0)","0(2)","1(5)","3(1)","4(3)","8(4)"],"object_location_counts":[{"shards":"0(2),1(5),3(1),4(3),8(4),13(0)","objects":19}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"1"},{"start":"27","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":93,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-11-01T04:17:21.186614+0000","last_change":"2023-11-01T04:17:14.626668+0000","last_active":"2023-11-01T04:17:21.186614+0000","last_peered":"2023-11-01T04:17:21.186614+0000","last_clean":"2023-11-01T04:17:21.186614+0000","last_became_active":"2023-11-01T04:16:01.930333+0000","last_became_peered":"2023-11-01T04:16:01.930333+0000","last_unstale":"2023-11-01T04:17:21.186614+0000","last_undegraded":"2023-11-01T04:17:21.186614+0000","last_fullsized":"2023-11-01T04:17:21.186614+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:17:14.626348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:17:14.626348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:18:29.583746+0000","scrub_duration":0.81031323899999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"99'87","reported_seq":362,"reported_epoch":99,"state":"active+undersized+degraded","last_fresh":"2023-11-01T04:17:21.714617+0000","last_change":"2023-11-01T04:17:00.175077+0000","last_active":"2023-11-01T04:17:21.714617+0000","last_peered":"2023-11-01T04:17:21.714617+0000","last_clean":"2023-11-01T04:16:58.345835+0000","last_became_active":"2023-11-01T04:17:00.175077+0000","last_became_peered":"2023-11-01T04:17:00.175077+0000","last_unstale":"2023-11-01T04:17:21.714617+0000","last_undegraded":"2023-11-01T04:16:59.870782+0000","last_fullsized":"2023-11-01T04:16:59.869770+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.38176358399999999,"stat_sum":{"num_bytes":9846784,"num_objects":16,"num_object_clones":11,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":50,"num_read_kb":11867,"num_write":87,"num_write_kb":9712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":7684096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9,1,2147483647,3,13],"acting":[6,4,9,1,2147483647,3,13],"avail_no_missing":["6(0)","1(3)","3(5)","4(1)","9(2)","13(6)"],"object_location_counts":[{"shards":"1(3),3(5),4(1),6(0),9(2),13(6)","objects":16}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"b","length":"1"},{"start":"15","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":99,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-11-01T04:17:22.197469+0000","last_change":"2023-11-01T04:17:01.990236+0000","last_active":"2023-11-01T04:17:22.197469+0000","last_peered":"2023-11-01T04:17:22.197469+0000","last_clean":"2023-11-01T04:17:22.197469+0000","last_became_active":"2023-11-01T04:16:53.814922+0000","last_became_peered":"2023-11-01T04:16:53.814922+0000","last_unstale":"2023-11-01T04:17:22.197469+0000","last_undegraded":"2023-11-01T04:17:22.197469+0000","last_fullsized":"2023-11-01T04:17:22.197469+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:17:01.989944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:17:01.989944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:18:10.387997+0000","scrub_duration":0.25018212899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14],"acting":[13,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"82'36","reported_seq":285,"reported_epoch":100,"state":"active+undersized+degraded","last_fresh":"2023-11-01T04:17:22.454267+0000","last_change":"2023-11-01T04:17:01.850256+0000","last_active":"2023-11-01T04:17:22.454267+0000","last_peered":"2023-11-01T04:17:22.454267+0000","last_clean":"2023-11-01T04:16:51.992246+0000","last_became_active":"2023-11-01T04:17:00.274425+0000","last_became_peered":"2023-11-01T04:17:00.274425+0000","last_unstale":"2023-11-01T04:17:22.454267+0000","last_undegraded":"2023-11-01T04:16:59.867003+0000","last_fullsized":"2023-11-01T04:16:59.866309+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20888748900000001,"stat_sum":{"num_bytes":6127616,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":7209,"num_write":19,"num_write_kb":5984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2703360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,4,1,10,15,6],"acting":[3,2147483647,4,1,10,15,6],"avail_no_missing":["3(0)","1(3)","4(2)","5(4)","6(6)","10(4)","15(5)"],"object_location_counts":[{"shards":"1(3),3(0),4(2),6(6),10(4),15(5)","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"b","length":"1"},{"start":"15","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":82,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-11-01T04:17:21.200477+0000","last_change":"2023-11-01T04:16:02.153588+0000","last_active":"2023-11-01T04:17:21.200477+0000","last_peered":"2023-11-01T04:17:21.200477+0000","last_clean":"2023-11-01T04:17:21.200477+0000","last_became_active":"2023-11-01T04:16:02.053124+0000","last_became_peered":"2023-11-01T04:16:02.053124+0000","last_unstale":"2023-11-01T04:17:21.200477+0000","last_undegraded":"2023-11-01T04:17:21.200477+0000","last_fullsized":"2023-11-01T04:17:21.200477+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"93'60","reported_seq":307,"reported_epoch":101,"state":"active+undersized+degraded","last_fresh":"2023-11-01T04:17:23.202016+0000","last_change":"2023-11-01T04:16:55.931609+0000","last_active":"2023-11-01T04:17:23.202016+0000","last_peered":"2023-11-01T04:17:23.202016+0000","last_clean":"2023-11-01T04:16:47.391101+0000","last_became_active":"2023-11-01T04:16:53.918374+0000","last_became_peered":"2023-11-01T04:16:53.918374+0000","last_unstale":"2023-11-01T04:17:23.202016+0000","last_undegraded":"2023-11-01T04:16:53.805895+0000","last_fullsized":"2023-11-01T04:16:53.804434+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:35.723802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063880600000000001,"stat_sum":{"num_bytes":7307264,"num_objects":11,"num_object_clones":8,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":41,"num_read_kb":25361,"num_write":62,"num_write_kb":7136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":8912896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,8,2,0,11,4,2147483647],"acting":[13,8,2,0,11,4,2147483647],"avail_no_missing":["13(0)","0(3)","2(2)","4(5)","5(5)","8(1)","11(4)"],"object_location_counts":[{"shards":"0(3),2(2),4(5),8(1),11(4),13(0)","objects":11}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.5","version":"96'34","reported_seq":289,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-11-01T04:17:22.097236+0000","last_change":"2023-11-01T04:17:19.072392+0000","last_active":"2023-11-01T04:17:22.097236+0000","last_peered":"2023-11-01T04:17:22.097236+0000","last_clean":"2023-11-01T04:17:22.097236+0000","last_became_active":"2023-11-01T04:16:11.168020+0000","last_became_peered":"2023-11-01T04:16:11.168020+0000","last_unstale":"2023-11-01T04:17:22.097236+0000","last_undegraded":"2023-11-01T04:17:22.097236+0000","last_fullsized":"2023-11-01T04:17:22.097236+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.40805799399999998,"stat_sum":{"num_bytes":2588672,"num_objects":4,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":6578,"num_write":7,"num_write_kb":2528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,14,13,10,2,11],"acting":[8,6,14,13,10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"4","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"2"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1b","length":"2"},{"start":"1e","length":"1"},{"start":"22","length":"1"},{"start":"26","length":"2"},{"start":"2f","length":"2"},{"start":"33","length":"1"},{"start":"38","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":82,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-11-01T04:17:21.110584+0000","last_change":"2023-11-01T04:17:15.120827+0000","last_active":"2023-11-01T04:17:21.110584+0000","last_peered":"2023-11-01T04:17:21.110584+0000","last_clean":"2023-11-01T04:17:21.110584+0000","last_became_active":"2023-11-01T04:16:15.173499+0000","last_became_peered":"2023-11-01T04:16:15.173499+0000","last_unstale":"2023-11-01T04:17:21.110584+0000","last_undegraded":"2023-11-01T04:17:21.110584+0000","last_fullsized":"2023-11-01T04:17:21.110584+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:17:15.120470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:17:15.120470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:18:16.464289+0000","scrub_duration":0.40526182300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.4","version":"96'81","reported_seq":352,"reported_epoch":100,"state":"active+clean","last_fresh":"2023-11-01T04:17:22.252237+0000","last_change":"2023-11-01T04:17:19.285471+0000","last_active":"2023-11-01T04:17:22.252237+0000","last_peered":"2023-11-01T04:17:22.252237+0000","last_clean":"2023-11-01T04:17:22.252237+0000","last_became_active":"2023-11-01T04:16:11.207236+0000","last_became_peered":"2023-11-01T04:16:11.207236+0000","last_unstale":"2023-11-01T04:17:22.252237+0000","last_undegraded":"2023-11-01T04:17:22.252237+0000","last_fullsized":"2023-11-01T04:17:22.252237+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:32.197055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003653707,"stat_sum":{"num_bytes":11567104,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":18827,"num_write":27,"num_write_kb":10928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,15,14,1,6,3,13],"acting":[10,15,14,1,6,3,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"4","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"2"},{"start":"15","length":"1"},{"start":"18","length":"1"},{"start":"1b","length":"2"},{"start":"1e","length":"1"},{"start":"22","length":"1"},{"start":"26","length":"2"},{"start":"2f","length":"2"},{"start":"33","length":"1"},{"start":"38","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":29,"reported_epoch":99,"state":"active+undersized","last_fresh":"2023-11-01T04:17:21.101228+0000","last_change":"2023-11-01T04:16:59.879292+0000","last_active":"2023-11-01T04:17:21.101228+0000","last_peered":"2023-11-01T04:17:21.101228+0000","last_clean":"2023-11-01T04:16:00.735965+0000","last_became_active":"2023-11-01T04:16:59.879292+0000","last_became_peered":"2023-11-01T04:16:59.879292+0000","last_unstale":"2023-11-01T04:17:21.101228+0000","last_undegraded":"2023-11-01T04:17:21.101228+0000","last_fullsized":"2023-11-01T04:16:59.873521+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:16:00.735965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:23.268660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"97'82","reported_seq":358,"reported_epoch":100,"state":"active+undersized+degraded","last_fresh":"2023-11-01T04:17:22.103012+0000","last_change":"2023-11-01T04:17:00.080241+0000","last_active":"2023-11-01T04:17:22.103012+0000","last_peered":"2023-11-01T04:17:22.103012+0000","last_clean":"2023-11-01T04:16:57.857423+0000","last_became_active":"2023-11-01T04:17:00.080241+0000","last_became_peered":"2023-11-01T04:17:00.080241+0000","last_unstale":"2023-11-01T04:17:22.103012+0000","last_undegraded":"2023-11-01T04:16:59.873581+0000","last_fullsized":"2023-11-01T04:16:59.871854+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:32.945414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037355090000000001,"stat_sum":{"num_bytes":8241152,"num_objects":10,"num_object_clones":6,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":16333,"num_write":53,"num_write_kb":8272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,9,2147483647,2,6,0],"acting":[14,11,9,2147483647,2,6,0],"avail_no_missing":["14(0)","0(6)","2(4)","6(5)","9(2)","11(1)"],"object_location_counts":[{"shards":"0(6),2(4),6(5),9(2),11(1),14(0)","objects":10}],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"4","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"},{"start":"15","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"1"},{"start":"27","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":90,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-11-01T04:17:21.475269+0000","last_change":"2023-11-01T04:17:06.482169+0000","last_active":"2023-11-01T04:17:21.475269+0000","last_peered":"2023-11-01T04:17:21.475269+0000","last_clean":"2023-11-01T04:17:21.475269+0000","last_became_active":"2023-11-01T04:16:01.875255+0000","last_became_peered":"2023-11-01T04:16:01.875255+0000","last_unstale":"2023-11-01T04:17:21.475269+0000","last_undegraded":"2023-11-01T04:17:21.475269+0000","last_fullsized":"2023-11-01T04:17:21.475269+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:17:06.481939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:17:06.481939+0000","last_clean_scrub_stamp":"2023-11-01T04:17:06.481939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:18:07.042956+0000","scrub_duration":0.51993433700000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11],"acting":[14,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.6","version":"100'30","reported_seq":242,"reported_epoch":100,"state":"active+undersized+degraded","last_fresh":"2023-11-01T04:17:22.516932+0000","last_change":"2023-11-01T04:16:48.993721+0000","last_active":"2023-11-01T04:17:22.516932+0000","last_peered":"2023-11-01T04:17:22.516932+0000","last_clean":"2023-11-01T04:16:46.979642+0000","last_became_active":"2023-11-01T04:16:48.993721+0000","last_became_peered":"2023-11-01T04:16:48.993721+0000","last_unstale":"2023-11-01T04:17:22.516932+0000","last_undegraded":"2023-11-01T04:16:48.786734+0000","last_fullsized":"2023-11-01T04:16:48.785840+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:09.864800+0000","last_clean_scrub_stamp":"2023-11-01T04:16:09.864800+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:17:25.165913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.106638213,"stat_sum":{"num_bytes":9453568,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":3655,"num_write":22,"num_write_kb":9232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,1,8,2147483647,0,9],"acting":[14,13,1,8,2147483647,0,9],"avail_no_missing":["14(0)","0(5)","1(2)","8(3)","9(6)","13(1)"],"object_location_counts":[{"shards":"0(5),1(2),8(3),9(6),13(1),14(0)","objects":6}],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"b","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":90,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-11-01T04:17:21.186762+0000","last_change":"2023-11-01T04:17:15.637503+0000","last_active":"2023-11-01T04:17:21.186762+0000","last_peered":"2023-11-01T04:17:21.186762+0000","last_clean":"2023-11-01T04:17:21.186762+0000","last_became_active":"2023-11-01T04:16:01.854811+0000","last_became_peered":"2023-11-01T04:16:01.854811+0000","last_unstale":"2023-11-01T04:17:21.186762+0000","last_undegraded":"2023-11-01T04:17:21.186762+0000","last_fullsized":"2023-11-01T04:17:21.186762+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T04:17:15.637201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T04:16:00.735965+0000","last_clean_scrub_stamp":"2023-11-01T04:17:15.637201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T04:18:26.702347+0000","scrub_duration":0.81986074600000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":149880832,"num_objects":162,"num_object_clones":112,"num_object_copies":1134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":132,"num_objects_misplaced":13,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":5,"num_read":486,"num_read_kb":209798,"num_write":770,"num_write_kb":156096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":29720576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249126912,"data_stored":395554816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1153,"ondisk_log_size":1153,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":15,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":22,"seq":94489280532,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34812,"kb_used_data":7784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337028,"statfs":{"total":96636764160,"available":96601116672,"internally_reserved":0,"allocated":7970816,"data_stored":9133016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,6,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.83399999999999996}]},{"osd":1,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":10.016,"5min":10.016,"15min":10.016},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":109.473,"5min":109.473,"15min":109.473},"last":0.91000000000000003},{"interface":"front","average":{"1min":3.9830000000000001,"5min":3.9830000000000001,"15min":3.9830000000000001},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":52.098999999999997,"5min":52.098999999999997,"15min":52.098999999999997},"last":1.093}]},{"osd":2,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.145,"15min":1.145},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":1.141},{"interface":"front","average":{"1min":10.115,"5min":10.115,"15min":10.115},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":109.532,"5min":109.532,"15min":109.532},"last":1.3720000000000001}]},{"osd":3,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.006},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.3540000000000001,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.3009999999999999}]},{"osd":4,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":13.137,"5min":13.137,"15min":13.137},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":109.26600000000001,"5min":109.26600000000001,"15min":109.26600000000001},"last":1.256},{"interface":"front","average":{"1min":7.9989999999999997,"5min":7.9989999999999997,"15min":7.9989999999999997},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":122.358,"5min":122.358,"15min":122.358},"last":1.2030000000000001}]},{"osd":5,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":2.5409999999999999,"5min":2.5409999999999999,"15min":2.5409999999999999},"last":1.534},{"interface":"front","average":{"1min":10.44,"5min":10.44,"15min":10.44},"min":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"max":{"1min":109.377,"5min":109.377,"15min":109.377},"last":1.73}]},{"osd":6,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":5.2220000000000004,"5min":5.2220000000000004,"15min":5.2220000000000004},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":69.561999999999998,"5min":69.561999999999998,"15min":69.561999999999998},"last":1.6839999999999999},{"interface":"front","average":{"1min":14.244999999999999,"5min":14.244999999999999,"15min":14.244999999999999},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":109.331,"5min":109.331,"15min":109.331},"last":1.8180000000000001}]},{"osd":7,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":11.640000000000001,"5min":11.640000000000001,"15min":11.640000000000001},"last":2.1200000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.4750000000000001,"5min":2.4750000000000001,"15min":2.4750000000000001},"last":1.7130000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":14,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":10.545,"5min":10.545,"15min":10.545},"min":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"max":{"1min":109.429,"5min":109.429,"15min":109.429},"last":1.649},{"interface":"front","average":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"min":{"1min":1.036,"5min":1.036,"15min":1.036},"max":{"1min":2.5409999999999999,"5min":2.5409999999999999,"15min":2.5409999999999999},"last":1.4119999999999999}]}]},{"osd":14,"up_from":22,"seq":94489280531,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46688,"kb_used_data":15508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325152,"statfs":{"total":96636764160,"available":96588955648,"internally_reserved":0,"allocated":15880192,"data_stored":22456760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 1 04:17:12 2023","interfaces":[{"interface":"back","average":{"1min":4.125,"5min":4.125,"15min":4.125},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":27.152999999999999,"5min":27.152999999999999,"15min":27.152999999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.86199999999999999}]},{"osd":1,"last update":"Wed Nov 1 04:17:00 2023","interfaces":[{"interface":"back","average":{"1min":6.4420000000000002,"5min":6.4420000000000002,"15min":6.4420000000000002},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":123.21899999999999,"5min":123.21899999999999,"15min":123.21899999999999},"last":1.069},{"interface":"front","average":{"1min":7.6589999999999998,"5min":7.6589999999999998,"15min":7.6589999999999998},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":95.117000000000004,"5min":95.117000000000004,"15min":95.117000000000004},"last":1.044}]},{"osd":2,"last update":"Wed Nov 1 04:17:00 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":1.01},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":16.262,"5min":16.262,"15min":16.262},"last":1.266}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.599}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.504}]},{"osd":8,"last update":"Wed Nov 1 04:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.996,"5min":1.996,"15min":1.996},"min":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"max":{"1min":16.350999999999999,"5min":16.350999999999999,"15min":16.350999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"max":{"1min":2.073,"5min":2.073,"15min":2.073},"last":1.159}]},{"osd":9,"last update":"Wed Nov 1 04:17:12 2023","interfaces":[{"interface":"back","average":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.0600000000000001},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":16.164000000000001,"5min":16.164000000000001,"15min":16.164000000000001},"last":1.504},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":1.292}]},{"osd":11,"last update":"Wed Nov 1 04:17:05 2023","interfaces":[{"interface":"back","average":{"1min":6.2130000000000001,"5min":6.2130000000000001,"15min":6.2130000000000001},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":46.173999999999999,"5min":46.173999999999999,"15min":46.173999999999999},"last":1.4039999999999999},{"interface":"front","average":{"1min":4.2690000000000001,"5min":4.2690000000000001,"15min":4.2690000000000001},"min":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":30.384,"5min":30.384,"15min":30.384},"last":1.2430000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]},{"osd":13,"last update":"Wed Nov 1 04:17:00 2023","interfaces":[{"interface":"back","average":{"1min":10.695,"5min":10.695,"15min":10.695},"min":{"1min":1.208,"5min":1.208,"15min":1.208},"max":{"1min":88.712999999999994,"5min":88.712999999999994,"15min":88.712999999999994},"last":1.3320000000000001},{"interface":"front","average":{"1min":2.222,"5min":2.222,"15min":2.222},"min":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"max":{"1min":16.306999999999999,"5min":16.306999999999999,"15min":16.306999999999999},"last":1.55}]},{"osd":15,"last update":"Wed Nov 1 04:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.417,"15min":1.417},"min":{"1min":1.091,"5min":1.091,"15min":1.091},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":1.367},{"interface":"front","average":{"1min":5.79,"5min":5.79,"15min":5.79},"min":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"max":{"1min":52.655000000000001,"5min":52.655000000000001,"15min":52.655000000000001},"last":1.4330000000000001}]}]},{"osd":13,"up_from":22,"seq":94489280532,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48780,"kb_used_data":17604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323060,"statfs":{"total":96636764160,"available":96586813440,"internally_reserved":0,"allocated":18026496,"data_stored":30598337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 1 04:17:14 2023","interfaces":[{"interface":"back","average":{"1min":22.904,"5min":22.904,"15min":22.904},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":259.661,"5min":259.661,"15min":259.661},"last":0.93999999999999995},{"interface":"front","average":{"1min":34.982999999999997,"5min":34.982999999999997,"15min":34.982999999999997},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":259.815,"5min":259.815,"15min":259.815},"last":1.2609999999999999}]},{"osd":1,"last update":"Wed Nov 1 04:16:58 2023","interfaces":[{"interface":"back","average":{"1min":6.8769999999999998,"5min":6.8769999999999998,"15min":6.8769999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":139.24100000000001,"5min":139.24100000000001,"15min":139.24100000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":22.045000000000002,"5min":22.045000000000002,"15min":22.045000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":259.60000000000002,"5min":259.60000000000002,"15min":259.60000000000002},"last":1.0800000000000001}]},{"osd":2,"last update":"Wed Nov 1 04:16:58 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":10.644,"5min":10.644,"15min":10.644},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":140,"5min":140,"15min":140},"last":0.84299999999999997}]},{"osd":3,"last update":"Wed Nov 1 04:16:58 2023","interfaces":[{"interface":"back","average":{"1min":11.526999999999999,"5min":11.526999999999999,"15min":11.526999999999999},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":139.90100000000001,"5min":139.90100000000001,"15min":139.90100000000001},"last":1.306},{"interface":"front","average":{"1min":27.177,"5min":27.177,"15min":27.177},"min":{"1min":0.876,"5min":0.876,"15min":0.876},"max":{"1min":259.74200000000002,"5min":259.74200000000002,"15min":259.74200000000002},"last":1.4810000000000001}]},{"osd":4,"last update":"Wed Nov 1 04:16:58 2023","interfaces":[{"interface":"back","average":{"1min":22.248000000000001,"5min":22.248000000000001,"15min":22.248000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":259.702,"5min":259.702,"15min":259.702},"last":1.619},{"interface":"front","average":{"1min":2.484,"5min":2.484,"15min":2.484},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":35.487000000000002,"5min":35.487000000000002,"15min":35.487000000000002},"last":1.512}]},{"osd":5,"last update":"Wed Nov 1 04:17:01 2023","interfaces":[{"interface":"back","average":{"1min":4.8799999999999999,"5min":4.8799999999999999,"15min":4.8799999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":44.689999999999998,"5min":44.689999999999998,"15min":44.689999999999998},"last":1.1899999999999999},{"interface":"front","average":{"1min":21.783999999999999,"5min":21.783999999999999,"15min":21.783999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":140.303,"5min":140.303,"15min":140.303},"last":1.4359999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]},{"osd":8,"last update":"Wed Nov 1 04:16:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.589},{"interface":"front","average":{"1min":7.4320000000000004,"5min":7.4320000000000004,"15min":7.4320000000000004},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":139.62299999999999,"5min":139.62299999999999,"15min":139.62299999999999},"last":1.552}]},{"osd":10,"last update":"Wed Nov 1 04:16:58 2023","interfaces":[{"interface":"back","average":{"1min":22.655000000000001,"5min":22.655000000000001,"15min":22.655000000000001},"min":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"max":{"1min":259.85300000000001,"5min":259.85300000000001,"15min":259.85300000000001},"last":1.79},{"interface":"front","average":{"1min":1.623,"5min":1.623,"15min":1.623},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":4.3159999999999998,"5min":4.3159999999999998,"15min":4.3159999999999998},"last":1.657}]},{"osd":11,"last update":"Wed Nov 1 04:17:14 2023","interfaces":[{"interface":"back","average":{"1min":27.625,"5min":27.625,"15min":27.625},"min":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"max":{"1min":259.779,"5min":259.779,"15min":259.779},"last":1.9359999999999999},{"interface":"front","average":{"1min":7.2569999999999997,"5min":7.2569999999999997,"15min":7.2569999999999997},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":94.640000000000001,"5min":94.640000000000001,"15min":94.640000000000001},"last":2.1139999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.784}]},{"osd":14,"last update":"Wed Nov 1 04:16:58 2023","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.544,"15min":1.544},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.3849999999999998,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":2.02},{"interface":"front","average":{"1min":7.6689999999999996,"5min":7.6689999999999996,"15min":7.6689999999999996},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":140.51300000000001,"5min":140.51300000000001,"15min":140.51300000000001},"last":1.7270000000000001}]},{"osd":15,"last update":"Wed Nov 1 04:16:58 2023","interfaces":[{"interface":"back","average":{"1min":10.217000000000001,"5min":10.217000000000001,"15min":10.217000000000001},"min":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"max":{"1min":199.55600000000001,"5min":199.55600000000001,"15min":199.55600000000001},"last":1.8420000000000001},{"interface":"front","average":{"1min":20.632000000000001,"5min":20.632000000000001,"15min":20.632000000000001},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":299.72399999999999,"5min":299.72399999999999,"15min":299.72399999999999},"last":1.8959999999999999}]}]},{"osd":6,"up_from":21,"seq":90194313237,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51668,"kb_used_data":20488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320172,"statfs":{"total":96636764160,"available":96583856128,"internally_reserved":0,"allocated":20979712,"data_stored":29368477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,3,4,5,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":291.661}]},{"osd":3,"last update":"Wed Nov 1 04:17:11 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.84999999999999998}]},{"osd":4,"last update":"Wed Nov 1 04:17:11 2023","interfaces":[{"interface":"back","average":{"1min":2.8239999999999998,"5min":2.8239999999999998,"15min":2.8239999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":43.716000000000001,"5min":43.716000000000001,"15min":43.716000000000001},"last":80.742000000000004},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.65200000000000002}]},{"osd":5,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":7.7039999999999997,"5min":7.7039999999999997,"15min":7.7039999999999997},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.97199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":81.061000000000007}]},{"osd":9,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.954,"5min":1.954,"15min":1.954},"last":1.0069999999999999},{"interface":"front","average":{"1min":3.1909999999999998,"5min":3.1909999999999998,"15min":3.1909999999999998},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":42.043999999999997,"5min":42.043999999999997,"15min":42.043999999999997},"last":1.0309999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":11,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":3.343,"5min":3.343,"15min":3.343},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":46.222999999999999,"5min":46.222999999999999,"15min":46.222999999999999},"last":80.905000000000001},{"interface":"front","average":{"1min":5.8479999999999999,"5min":5.8479999999999999,"15min":5.8479999999999999},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":93.313000000000002,"5min":93.313000000000002,"15min":93.313000000000002},"last":1.133}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":13,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":7.4580000000000002,"5min":7.4580000000000002,"15min":7.4580000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":70.855999999999995,"5min":70.855999999999995,"15min":70.855999999999995},"last":80.856999999999999},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":1.1819999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1899999999999999}]},{"osd":15,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":2.5529999999999999,"5min":2.5529999999999999,"15min":2.5529999999999999},"last":210.654},{"interface":"front","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":1.728,"5min":1.728,"15min":1.728},"last":1.105}]}]},{"osd":0,"up_from":22,"seq":94489280532,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52048,"kb_used_data":20868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319792,"statfs":{"total":96636764160,"available":96583467008,"internally_reserved":0,"allocated":21368832,"data_stored":36153784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":15.279,"5min":15.279,"15min":15.279},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":179.001,"5min":179.001,"15min":179.001},"last":0.79500000000000004},{"interface":"front","average":{"1min":5.4619999999999997,"5min":5.4619999999999997,"15min":5.4619999999999997},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":76.783000000000001,"5min":76.783000000000001,"15min":76.783000000000001},"last":0.95599999999999996}]},{"osd":2,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":4.5759999999999996,"5min":4.5759999999999996,"15min":4.5759999999999996},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":76.706999999999994,"5min":76.706999999999994,"15min":76.706999999999994},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.85799999999999998}]},{"osd":3,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":14.462999999999999,"5min":14.462999999999999,"15min":14.462999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":179.47900000000001,"5min":179.47900000000001,"15min":179.47900000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":4.6909999999999998,"5min":4.6909999999999998,"15min":4.6909999999999998},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":76.820999999999998,"5min":76.820999999999998,"15min":76.820999999999998},"last":0.749}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]},{"osd":6,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":3.7490000000000001,"5min":3.7490000000000001,"15min":3.7490000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":59.93,"5min":59.93,"15min":59.93},"last":1.038},{"interface":"front","average":{"1min":16.23,"5min":16.23,"15min":16.23},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":179.31399999999999,"5min":179.31399999999999,"15min":179.31399999999999},"last":1.097}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":9,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":4.8010000000000002,"5min":4.8010000000000002,"15min":4.8010000000000002},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":76.905000000000001,"5min":76.905000000000001,"15min":76.905000000000001},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.956,"5min":1.956,"15min":1.956},"last":0.98999999999999999}]},{"osd":10,"last update":"Wed Nov 1 04:17:11 2023","interfaces":[{"interface":"back","average":{"1min":9.093,"5min":9.093,"15min":9.093},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":76.983999999999995,"5min":76.983999999999995,"15min":76.983999999999995},"last":1.1679999999999999},{"interface":"front","average":{"1min":9.2569999999999997,"5min":9.2569999999999997,"15min":9.2569999999999997},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":77.063999999999993,"5min":77.063999999999993,"15min":77.063999999999993},"last":1.264}]},{"osd":11,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":17.024000000000001,"5min":17.024000000000001,"15min":17.024000000000001},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":179.63499999999999,"5min":179.63499999999999,"15min":179.63499999999999},"last":1.206},{"interface":"front","average":{"1min":5.0279999999999996,"5min":5.0279999999999996,"15min":5.0279999999999996},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":77.022000000000006,"5min":77.022000000000006,"15min":77.022000000000006},"last":1.3240000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":13,"last update":"Wed Nov 1 04:17:11 2023","interfaces":[{"interface":"back","average":{"1min":11.231,"5min":11.231,"15min":11.231},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":90.777000000000001,"5min":90.777000000000001,"15min":90.777000000000001},"last":1.403},{"interface":"front","average":{"1min":10.548999999999999,"5min":10.548999999999999,"15min":10.548999999999999},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":90.712999999999994,"5min":90.712999999999994,"15min":90.712999999999994},"last":1.143}]},{"osd":14,"last update":"Wed Nov 1 04:17:11 2023","interfaces":[{"interface":"back","average":{"1min":16.140999999999998,"5min":16.140999999999998,"15min":16.140999999999998},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":179.55600000000001,"5min":179.55600000000001,"15min":179.55600000000001},"last":1.294},{"interface":"front","average":{"1min":17.797999999999998,"5min":17.797999999999998,"15min":17.797999999999998},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":181.12700000000001,"5min":181.12700000000001,"15min":181.12700000000001},"last":1.4470000000000001}]},{"osd":15,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.248,"15min":1.248},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.5979999999999999,"5min":2.5979999999999999,"15min":2.5979999999999999},"last":1.238},{"interface":"front","average":{"1min":14.879,"5min":14.879,"15min":14.879},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":179.70500000000001,"5min":179.70500000000001,"15min":179.70500000000001},"last":1.367}]}]},{"osd":1,"up_from":22,"seq":94489280532,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56064,"kb_used_data":24888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94315776,"statfs":{"total":96636764160,"available":96579354624,"internally_reserved":0,"allocated":25485312,"data_stored":38721976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 1 04:17:01 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":58.667999999999999,"5min":58.667999999999999,"15min":58.667999999999999},"last":0.70399999999999996}]},{"osd":2,"last update":"Wed Nov 1 04:17:01 2023","interfaces":[{"interface":"back","average":{"1min":8.391,"5min":8.391,"15min":8.391},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":84.043000000000006,"5min":84.043000000000006,"15min":84.043000000000006},"last":0.76800000000000002},{"interface":"front","average":{"1min":4.6760000000000002,"5min":4.6760000000000002,"15min":4.6760000000000002},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":55.851999999999997,"5min":55.851999999999997,"15min":55.851999999999997},"last":0.92900000000000005}]},{"osd":3,"last update":"Wed Nov 1 04:17:01 2023","interfaces":[{"interface":"back","average":{"1min":22.364999999999998,"5min":22.364999999999998,"15min":22.364999999999998},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":196.58500000000001,"5min":196.58500000000001,"15min":196.58500000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":9.7789999999999999,"5min":9.7789999999999999,"15min":9.7789999999999999},"last":0.86199999999999999}]},{"osd":4,"last update":"Wed Nov 1 04:17:01 2023","interfaces":[{"interface":"back","average":{"1min":9.8409999999999993,"5min":9.8409999999999993,"15min":9.8409999999999993},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":72.451999999999998,"5min":72.451999999999998,"15min":72.451999999999998},"last":0.84899999999999998},{"interface":"front","average":{"1min":4.516,"5min":4.516,"15min":4.516},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":72.420000000000002,"5min":72.420000000000002,"15min":72.420000000000002},"last":0.89500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":6,"last update":"Wed Nov 1 04:17:01 2023","interfaces":[{"interface":"back","average":{"1min":5.5720000000000001,"5min":5.5720000000000001,"15min":5.5720000000000001},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":90.807000000000002,"5min":90.807000000000002,"15min":90.807000000000002},"last":0.80700000000000005},{"interface":"front","average":{"1min":9.4429999999999996,"5min":9.4429999999999996,"15min":9.4429999999999996},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":60.317999999999998,"5min":60.317999999999998,"15min":60.317999999999998},"last":0.98999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":8,"last update":"Wed Nov 1 04:17:01 2023","interfaces":[{"interface":"back","average":{"1min":4.9080000000000004,"5min":4.9080000000000004,"15min":4.9080000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":55.527999999999999,"5min":55.527999999999999,"15min":55.527999999999999},"last":1.0449999999999999},{"interface":"front","average":{"1min":22.050999999999998,"5min":22.050999999999998,"15min":22.050999999999998},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":111.149,"5min":111.149,"15min":111.149},"last":0.97399999999999998}]},{"osd":9,"last update":"Wed Nov 1 04:17:01 2023","interfaces":[{"interface":"back","average":{"1min":5.0750000000000002,"5min":5.0750000000000002,"15min":5.0750000000000002},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":55.762999999999998,"5min":55.762999999999998,"15min":55.762999999999998},"last":0.874},{"interface":"front","average":{"1min":12.499000000000001,"5min":12.499000000000001,"15min":12.499000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":118.92,"5min":118.92,"15min":118.92},"last":0.90700000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":14,"last update":"Wed Nov 1 04:17:01 2023","interfaces":[{"interface":"back","average":{"1min":21.07,"5min":21.07,"15min":21.07},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":201.12299999999999,"5min":201.12299999999999,"15min":201.12299999999999},"last":1.135},{"interface":"front","average":{"1min":5.1479999999999997,"5min":5.1479999999999997,"15min":5.1479999999999997},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":56.079999999999998,"5min":56.079999999999998,"15min":56.079999999999998},"last":1.173}]}]},{"osd":2,"up_from":22,"seq":94489280533,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40612,"kb_used_data":13580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331228,"statfs":{"total":96636764160,"available":96595177472,"internally_reserved":0,"allocated":13905920,"data_stored":21608888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":4.1760000000000002,"5min":4.1760000000000002,"15min":4.1760000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":73.019000000000005,"5min":73.019000000000005,"15min":73.019000000000005},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":2.246,"5min":2.246,"15min":2.246},"last":0.80100000000000005}]},{"osd":3,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":4.1790000000000003,"5min":4.1790000000000003,"15min":4.1790000000000003},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":70.722999999999999,"5min":70.722999999999999,"15min":70.722999999999999},"last":0.872},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.962,"5min":1.962,"15min":1.962},"last":1.0229999999999999}]},{"osd":4,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.95199999999999996},{"interface":"front","average":{"1min":4.0940000000000003,"5min":4.0940000000000003,"15min":4.0940000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":70.558999999999997,"5min":70.558999999999997,"15min":70.558999999999997},"last":0.93799999999999994}]},{"osd":5,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.2280000000000002,"5min":2.2280000000000002,"15min":2.2280000000000002},"last":0.85199999999999998},{"interface":"front","average":{"1min":4.1360000000000001,"5min":4.1360000000000001,"15min":4.1360000000000001},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":70.808000000000007,"5min":70.808000000000007,"15min":70.808000000000007},"last":0.75900000000000001}]},{"osd":6,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":5.4640000000000004,"5min":5.4640000000000004,"15min":5.4640000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":100.395,"5min":100.395,"15min":100.395},"last":0.91300000000000003},{"interface":"front","average":{"1min":5.7060000000000004,"5min":5.7060000000000004,"15min":5.7060000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":100.726,"5min":100.726,"15min":100.726},"last":0.65000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]},{"osd":8,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":4.4290000000000003,"5min":4.4290000000000003,"15min":4.4290000000000003},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":70.244,"5min":70.244,"15min":70.244},"last":0.89300000000000002},{"interface":"front","average":{"1min":4.2619999999999996,"5min":4.2619999999999996,"15min":4.2619999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":70.643000000000001,"5min":70.643000000000001,"15min":70.643000000000001},"last":0.96999999999999997}]},{"osd":9,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.298,"15min":1.298},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.4510000000000001,"5min":2.4510000000000001,"15min":2.4510000000000001},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":2.6819999999999999,"5min":2.6819999999999999,"15min":2.6819999999999999},"last":0.98799999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3859999999999999}]},{"osd":13,"last update":"Wed Nov 1 04:16:58 2023","interfaces":[{"interface":"back","average":{"1min":5.6189999999999998,"5min":5.6189999999999998,"15min":5.6189999999999998},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":94.789000000000001,"5min":94.789000000000001,"15min":94.789000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":3.9550000000000001,"5min":3.9550000000000001,"15min":3.9550000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":57.752000000000002,"5min":57.752000000000002,"15min":57.752000000000002},"last":1.081}]},{"osd":15,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":4.4470000000000001,"5min":4.4470000000000001,"15min":4.4470000000000001},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":70.864000000000004,"5min":70.864000000000004,"15min":70.864000000000004},"last":1.224},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.552,"5min":2.552,"15min":2.552},"last":1.1459999999999999}]}]},{"osd":3,"up_from":22,"seq":94489280532,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59104,"kb_used_data":27924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94312736,"statfs":{"total":96636764160,"available":96576241664,"internally_reserved":0,"allocated":28594176,"data_stored":43037345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4359999999999999}]},{"osd":1,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.345},{"interface":"front","average":{"1min":2.4889999999999999,"5min":2.4889999999999999,"15min":2.4889999999999999},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":29.544,"5min":29.544,"15min":29.544},"last":0.92900000000000005}]},{"osd":2,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":6.1219999999999999,"5min":6.1219999999999999,"15min":6.1219999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":54.049999999999997,"5min":54.049999999999997,"15min":54.049999999999997},"last":1.375},{"interface":"front","average":{"1min":3.7650000000000001,"5min":3.7650000000000001,"15min":3.7650000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":54.091000000000001,"5min":54.091000000000001,"15min":54.091000000000001},"last":1.25}]},{"osd":4,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":3.2559999999999998,"5min":3.2559999999999998,"15min":3.2559999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":47.768000000000001,"5min":47.768000000000001,"15min":47.768000000000001},"last":0.78400000000000003}]},{"osd":5,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":2.7589999999999999,"5min":2.7589999999999999,"15min":2.7589999999999999},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":33.008000000000003,"5min":33.008000000000003,"15min":33.008000000000003},"last":1.1060000000000001},{"interface":"front","average":{"1min":3.4239999999999999,"5min":3.4239999999999999,"15min":3.4239999999999999},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":47.843000000000004,"5min":47.843000000000004,"15min":47.843000000000004},"last":0.97499999999999998}]},{"osd":6,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.137,"15min":1.137},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.6200000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":1.1619999999999999},{"interface":"front","average":{"1min":5.5519999999999996,"5min":5.5519999999999996,"15min":5.5519999999999996},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":87.691999999999993,"5min":87.691999999999993,"15min":87.691999999999993},"last":1.4059999999999999}]},{"osd":7,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":2.9870000000000001,"5min":2.9870000000000001,"15min":2.9870000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":38.767000000000003,"5min":38.767000000000003,"15min":38.767000000000003},"last":1.405},{"interface":"front","average":{"1min":2.903,"5min":2.903,"15min":2.903},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":32.869999999999997,"5min":32.869999999999997,"15min":32.869999999999997},"last":32.869999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2809999999999999}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1970000000000001}]},{"osd":10,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":6.2389999999999999,"5min":6.2389999999999999,"15min":6.2389999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":76.897999999999996,"5min":76.897999999999996,"15min":76.897999999999996},"last":56.627000000000002},{"interface":"front","average":{"1min":8.1950000000000003,"5min":8.1950000000000003,"15min":8.1950000000000003},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":83.721000000000004,"5min":83.721000000000004,"15min":83.721000000000004},"last":57.509999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3169999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.623}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.526}]},{"osd":15,"last update":"Wed Nov 1 04:17:11 2023","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.645,"15min":3.645},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":47.716999999999999,"5min":47.716999999999999,"15min":47.716999999999999},"last":1.468},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":1.552}]}]},{"osd":4,"up_from":22,"seq":94489280532,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58064,"kb_used_data":26884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94313776,"statfs":{"total":96636764160,"available":96577306624,"internally_reserved":0,"allocated":27529216,"data_stored":44208801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":3,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":5.7599999999999998,"5min":5.7599999999999998,"15min":5.7599999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":98.561999999999998,"5min":98.561999999999998,"15min":98.561999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":14.195,"5min":14.195,"15min":14.195},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":208.64500000000001,"5min":208.64500000000001,"15min":208.64500000000001},"last":0.68600000000000005}]},{"osd":5,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":14.689,"5min":14.689,"15min":14.689},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":145.809,"5min":145.809,"15min":145.809},"last":0.76400000000000001},{"interface":"front","average":{"1min":25.039999999999999,"5min":25.039999999999999,"15min":25.039999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":208.59999999999999,"5min":208.59999999999999,"15min":208.59999999999999},"last":0.60299999999999998}]},{"osd":6,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":90.921999999999997,"5min":90.921999999999997,"15min":90.921999999999997},"last":0.626},{"interface":"front","average":{"1min":10.597,"5min":10.597,"15min":10.597},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":102.577,"5min":102.577,"15min":102.577},"last":0.83799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7689999999999999}]},{"osd":8,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.833,"5min":1.833,"15min":1.833},"last":0.65000000000000002},{"interface":"front","average":{"1min":14.249000000000001,"5min":14.249000000000001,"15min":14.249000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":208.68799999999999,"5min":208.68799999999999,"15min":208.68799999999999},"last":0.94199999999999995}]},{"osd":9,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":7.6230000000000002,"5min":7.6230000000000002,"15min":7.6230000000000002},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":78.045000000000002,"5min":78.045000000000002,"15min":78.045000000000002},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.214,"5min":1.214,"15min":1.214},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.95899999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":51.167000000000002}]},{"osd":11,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":8.7539999999999996,"5min":8.7539999999999996,"15min":8.7539999999999996},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":78.025000000000006,"5min":78.025000000000006,"15min":78.025000000000006},"last":0.67000000000000004},{"interface":"front","average":{"1min":6.0060000000000002,"5min":6.0060000000000002,"15min":6.0060000000000002},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":46.317999999999998,"5min":46.317999999999998,"15min":46.317999999999998},"last":0.875}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":13,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":29.454000000000001,"5min":29.454000000000001,"15min":29.454000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":299.48099999999999,"5min":299.48099999999999,"15min":299.48099999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":22.66,"5min":22.66,"15min":22.66},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":299.42500000000001,"5min":299.42500000000001,"15min":299.42500000000001},"last":1.1160000000000001}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]}]},{"osd":5,"up_from":0,"seq":94489280532,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":7,"up_from":22,"seq":94489280525,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36096,"kb_used_data":9076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335744,"statfs":{"total":96636764160,"available":96599801856,"internally_reserved":0,"allocated":9293824,"data_stored":14943437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,6,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0589999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3029999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.585}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5129999999999999}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3120000000000001}]}]},{"osd":8,"up_from":22,"seq":94489280533,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44816,"kb_used_data":17800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327024,"statfs":{"total":96636764160,"available":96590872576,"internally_reserved":0,"allocated":18227200,"data_stored":30771640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.6719999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":9.734,"5min":9.734,"15min":9.734},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":98.290999999999997,"5min":98.290999999999997,"15min":98.290999999999997},"last":0.96899999999999997}]},{"osd":2,"last update":"Wed Nov 1 04:17:11 2023","interfaces":[{"interface":"back","average":{"1min":16.812000000000001,"5min":16.812000000000001,"15min":16.812000000000001},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":155.41200000000001,"5min":155.41200000000001,"15min":155.41200000000001},"last":1.24},{"interface":"front","average":{"1min":9.0579999999999998,"5min":9.0579999999999998,"15min":9.0579999999999998},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":98.376000000000005,"5min":98.376000000000005,"15min":98.376000000000005},"last":0.84399999999999997}]},{"osd":3,"last update":"Wed Nov 1 04:17:01 2023","interfaces":[{"interface":"back","average":{"1min":13.18,"5min":13.18,"15min":13.18},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":98.584000000000003,"5min":98.584000000000003,"15min":98.584000000000003},"last":1.0489999999999999},{"interface":"front","average":{"1min":8.6159999999999997,"5min":8.6159999999999997,"15min":8.6159999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":156.648,"5min":156.648,"15min":156.648},"last":1.0740000000000001}]},{"osd":6,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":11.865,"5min":11.865,"15min":11.865},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":109.11499999999999,"5min":109.11499999999999,"15min":109.11499999999999},"last":1.286},{"interface":"front","average":{"1min":6.3529999999999998,"5min":6.3529999999999998,"15min":6.3529999999999998},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":108.42400000000001,"5min":108.42400000000001,"15min":108.42400000000001},"last":1.167}]},{"osd":7,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":23.405000000000001,"5min":23.405000000000001,"15min":23.405000000000001},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":197.03700000000001,"5min":197.03700000000001,"15min":197.03700000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":14.621,"5min":14.621,"15min":14.621},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":197.05699999999999,"5min":197.05699999999999,"15min":197.05699999999999},"last":0.71699999999999997}]},{"osd":9,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":5.4359999999999999,"5min":5.4359999999999999,"15min":5.4359999999999999},"last":0.59699999999999998}]},{"osd":10,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":4.3449999999999998,"5min":4.3449999999999998,"15min":4.3449999999999998},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":67.665999999999997,"5min":67.665999999999997,"15min":67.665999999999997},"last":0.94099999999999995},{"interface":"front","average":{"1min":4.9630000000000001,"5min":4.9630000000000001,"15min":4.9630000000000001},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":79.281000000000006,"5min":79.281000000000006,"15min":79.281000000000006},"last":1.103}]},{"osd":11,"last update":"Wed Nov 1 04:17:11 2023","interfaces":[{"interface":"back","average":{"1min":13.039999999999999,"5min":13.039999999999999,"15min":13.039999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":98.647999999999996,"5min":98.647999999999996,"15min":98.647999999999996},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":5.4100000000000001,"5min":5.4100000000000001,"15min":5.4100000000000001},"last":0.89000000000000001}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":13,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":31.158999999999999,"5min":31.158999999999999,"15min":31.158999999999999},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":208.90000000000001,"5min":208.90000000000001,"15min":208.90000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.651,"5min":1.651,"15min":1.651},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":11.507999999999999,"5min":11.507999999999999,"15min":11.507999999999999},"last":0.999}]},{"osd":14,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":17.908000000000001,"5min":17.908000000000001,"15min":17.908000000000001},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":192.94499999999999,"5min":192.94499999999999,"15min":192.94499999999999},"last":1.026},{"interface":"front","average":{"1min":16.388999999999999,"5min":16.388999999999999,"15min":16.388999999999999},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":98.209999999999994,"5min":98.209999999999994,"15min":98.209999999999994},"last":86.534999999999997}]},{"osd":15,"last update":"Wed Nov 1 04:16:57 2023","interfaces":[{"interface":"back","average":{"1min":10.887,"5min":10.887,"15min":10.887},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":98.718000000000004,"5min":98.718000000000004,"15min":98.718000000000004},"last":1.196},{"interface":"front","average":{"1min":4.569,"5min":4.569,"15min":4.569},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":66.412999999999997,"5min":66.412999999999997,"15min":66.412999999999997},"last":1.1240000000000001}]}]},{"osd":9,"up_from":22,"seq":94489280532,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40524,"kb_used_data":13504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331316,"statfs":{"total":96636764160,"available":96595267584,"internally_reserved":0,"allocated":13828096,"data_stored":19309217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[3,4,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":12.182,"5min":12.182,"15min":12.182},"last":1.0089999999999999},{"interface":"front","average":{"1min":7.4720000000000004,"5min":7.4720000000000004,"15min":7.4720000000000004},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":119.896,"5min":119.896,"15min":119.896},"last":0.79500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":6,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":3.7869999999999999,"5min":3.7869999999999999,"15min":3.7869999999999999},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":48.270000000000003,"5min":48.270000000000003,"15min":48.270000000000003},"last":0.83999999999999997},{"interface":"front","average":{"1min":10.295,"5min":10.295,"15min":10.295},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":119.59,"5min":119.59,"15min":119.59},"last":1.173}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":8,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.629},{"interface":"front","average":{"1min":4.657,"5min":4.657,"15min":4.657},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":47.716999999999999,"5min":47.716999999999999,"15min":47.716999999999999},"last":0.58199999999999996}]},{"osd":10,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":10.180999999999999,"5min":10.180999999999999,"15min":10.180999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":119.749,"5min":119.749,"15min":119.749},"last":1.25}]},{"osd":11,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":3.698,"5min":3.698,"15min":3.698},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":44.148000000000003,"5min":44.148000000000003,"15min":44.148000000000003},"last":0.76100000000000001},{"interface":"front","average":{"1min":27.292999999999999,"5min":27.292999999999999,"15min":27.292999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":291.06700000000001,"5min":291.06700000000001,"15min":291.06700000000001},"last":0.92100000000000004}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":13,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":10.488,"5min":10.488,"15min":10.488},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":98.363,"5min":98.363,"15min":98.363},"last":0.874},{"interface":"front","average":{"1min":2.1739999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":1.052}]},{"osd":14,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":21.279,"5min":21.279,"15min":21.279},"min":{"1min":1.022,"5min":1.022,"15min":1.022},"max":{"1min":119.798,"5min":119.798,"15min":119.798},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"min":{"1min":0.878,"5min":0.878,"15min":0.878},"max":{"1min":2.8809999999999998,"5min":2.8809999999999998,"15min":2.8809999999999998},"last":1.085}]},{"osd":15,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"min":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"max":{"1min":3.3159999999999998,"5min":3.3159999999999998,"15min":3.3159999999999998},"last":0.96199999999999997},{"interface":"front","average":{"1min":8.0549999999999997,"5min":8.0549999999999997,"15min":8.0549999999999997},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":119.67100000000001,"5min":119.67100000000001,"15min":119.67100000000001},"last":1.3100000000000001}]}]},{"osd":10,"up_from":22,"seq":94489280532,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40360,"kb_used_data":13344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331480,"statfs":{"total":96636764160,"available":96595435520,"internally_reserved":0,"allocated":13664256,"data_stored":18512312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":1,"last update":"Wed Nov 1 04:16:58 2023","interfaces":[{"interface":"back","average":{"1min":12.068,"5min":12.068,"15min":12.068},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":94.176000000000002,"5min":94.176000000000002,"15min":94.176000000000002},"last":0.85499999999999998},{"interface":"front","average":{"1min":7.319,"5min":7.319,"15min":7.319},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":87.129999999999995,"5min":87.129999999999995,"15min":87.129999999999995},"last":1.0469999999999999}]},{"osd":3,"last update":"Wed Nov 1 04:17:15 2023","interfaces":[{"interface":"back","average":{"1min":5.1159999999999997,"5min":5.1159999999999997,"15min":5.1159999999999997},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":77.810000000000002,"5min":77.810000000000002,"15min":77.810000000000002},"last":0.78400000000000003},{"interface":"front","average":{"1min":8.0879999999999992,"5min":8.0879999999999992,"15min":8.0879999999999992},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":87.340999999999994,"5min":87.340999999999994,"15min":87.340999999999994},"last":0.81499999999999995}]},{"osd":4,"last update":"Wed Nov 1 04:17:15 2023","interfaces":[{"interface":"back","average":{"1min":7.367,"5min":7.367,"15min":7.367},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":85.304000000000002,"5min":85.304000000000002,"15min":85.304000000000002},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":14.545999999999999,"5min":14.545999999999999,"15min":14.545999999999999},"last":0.871}]},{"osd":6,"last update":"Wed Nov 1 04:17:15 2023","interfaces":[{"interface":"back","average":{"1min":11.288,"5min":11.288,"15min":11.288},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":129.988,"5min":129.988,"15min":129.988},"last":0.69799999999999995},{"interface":"front","average":{"1min":4.2830000000000004,"5min":4.2830000000000004,"15min":4.2830000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":56.893000000000001,"5min":56.893000000000001,"15min":56.893000000000001},"last":0.73899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":59.899000000000001}]},{"osd":8,"last update":"Wed Nov 1 04:16:58 2023","interfaces":[{"interface":"back","average":{"1min":4.9539999999999997,"5min":4.9539999999999997,"15min":4.9539999999999997},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":87.620000000000005,"5min":87.620000000000005,"15min":87.620000000000005},"last":0.88800000000000001},{"interface":"front","average":{"1min":3.7389999999999999,"5min":3.7389999999999999,"15min":3.7389999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":57.042999999999999,"5min":57.042999999999999,"15min":57.042999999999999},"last":1.1539999999999999}]},{"osd":9,"last update":"Wed Nov 1 04:16:58 2023","interfaces":[{"interface":"back","average":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":14.935,"5min":14.935,"15min":14.935},"last":0.91400000000000003},{"interface":"front","average":{"1min":3.9209999999999998,"5min":3.9209999999999998,"15min":3.9209999999999998},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":56.947000000000003,"5min":56.947000000000003,"15min":56.947000000000003},"last":1.276}]},{"osd":11,"last update":"Wed Nov 1 04:16:58 2023","interfaces":[{"interface":"back","average":{"1min":6.056,"5min":6.056,"15min":6.056},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":96.762,"5min":96.762,"15min":96.762},"last":0.99199999999999999},{"interface":"front","average":{"1min":4.6500000000000004,"5min":4.6500000000000004,"15min":4.6500000000000004},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":57.133000000000003,"5min":57.133000000000003,"15min":57.133000000000003},"last":1.234}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3200000000000001}]},{"osd":13,"last update":"Wed Nov 1 04:16:58 2023","interfaces":[{"interface":"back","average":{"1min":8.9749999999999996,"5min":8.9749999999999996,"15min":8.9749999999999996},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":89.876999999999995,"5min":89.876999999999995,"15min":89.876999999999995},"last":1.028},{"interface":"front","average":{"1min":18.481000000000002,"5min":18.481000000000002,"15min":18.481000000000002},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":169.89099999999999,"5min":169.89099999999999,"15min":169.89099999999999},"last":1.0109999999999999}]},{"osd":14,"last update":"Wed Nov 1 04:16:58 2023","interfaces":[{"interface":"back","average":{"1min":5.3170000000000002,"5min":5.3170000000000002,"15min":5.3170000000000002},"min":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"max":{"1min":87.495999999999995,"5min":87.495999999999995,"15min":87.495999999999995},"last":1.1950000000000001},{"interface":"front","average":{"1min":7.2439999999999998,"5min":7.2439999999999998,"15min":7.2439999999999998},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":62.985999999999997,"5min":62.985999999999997,"15min":62.985999999999997},"last":1.3220000000000001}]},{"osd":15,"last update":"Wed Nov 1 04:16:58 2023","interfaces":[{"interface":"back","average":{"1min":2.105,"5min":2.105,"15min":2.105},"min":{"1min":1.048,"5min":1.048,"15min":1.048},"max":{"1min":15.169,"5min":15.169,"15min":15.169},"last":1.133},{"interface":"front","average":{"1min":5.3019999999999996,"5min":5.3019999999999996,"15min":5.3019999999999996},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":87.539000000000001,"5min":87.539000000000001,"15min":87.539000000000001},"last":1.0960000000000001}]}]},{"osd":11,"up_from":22,"seq":94489280531,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52016,"kb_used_data":20836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319824,"statfs":{"total":96636764160,"available":96583499776,"internally_reserved":0,"allocated":21336064,"data_stored":32065976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]},{"osd":1,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":3.262,"5min":3.262,"15min":3.262},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":40.642000000000003,"5min":40.642000000000003,"15min":40.642000000000003},"last":0.84499999999999997},{"interface":"front","average":{"1min":5.5259999999999998,"5min":5.5259999999999998,"15min":5.5259999999999998},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":47.341000000000001,"5min":47.341000000000001,"15min":47.341000000000001},"last":0.78400000000000003}]},{"osd":2,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":3.3799999999999999,"5min":3.3799999999999999,"15min":3.3799999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":40.698999999999998,"5min":40.698999999999998,"15min":40.698999999999998},"last":1.268},{"interface":"front","average":{"1min":11.314,"5min":11.314,"15min":11.314},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":75.811999999999998,"5min":75.811999999999998,"15min":75.811999999999998},"last":0.95699999999999996}]},{"osd":3,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":3.2869999999999999,"5min":3.2869999999999999,"15min":3.2869999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":40.454000000000001,"5min":40.454000000000001,"15min":40.454000000000001},"last":1.1319999999999999},{"interface":"front","average":{"1min":11.222,"5min":11.222,"15min":11.222},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":75.653000000000006,"5min":75.653000000000006,"15min":75.653000000000006},"last":0.71699999999999997}]},{"osd":4,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":5.6479999999999997,"5min":5.6479999999999997,"15min":5.6479999999999997},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":47.167000000000002,"5min":47.167000000000002,"15min":47.167000000000002},"last":50.691000000000003},{"interface":"front","average":{"1min":11.333,"5min":11.333,"15min":11.333},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":75.707999999999998,"5min":75.707999999999998,"15min":75.707999999999998},"last":91.414000000000001}]},{"osd":5,"last update":"Wed Nov 1 04:17:13 2023","interfaces":[{"interface":"back","average":{"1min":12.663,"5min":12.663,"15min":12.663},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":75.790999999999997,"5min":75.790999999999997,"15min":75.790999999999997},"last":1.0629999999999999},{"interface":"front","average":{"1min":3.7149999999999999,"5min":3.7149999999999999,"15min":3.7149999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":40.738999999999997,"5min":40.738999999999997,"15min":40.738999999999997},"last":1.365}]},{"osd":9,"last update":"Wed Nov 1 04:17:13 2023","interfaces":[{"interface":"back","average":{"1min":10.43,"5min":10.43,"15min":10.43},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":68.122,"5min":68.122,"15min":68.122},"last":0.90700000000000003},{"interface":"front","average":{"1min":12.616,"5min":12.616,"15min":12.616},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":75.840000000000003,"5min":75.840000000000003,"15min":75.840000000000003},"last":52.213999999999999}]},{"osd":10,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":7.3849999999999998,"5min":7.3849999999999998,"15min":7.3849999999999998},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":66.361999999999995,"5min":66.361999999999995,"15min":66.361999999999995},"last":1.3180000000000001},{"interface":"front","average":{"1min":9.6099999999999994,"5min":9.6099999999999994,"15min":9.6099999999999994},"min":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"max":{"1min":66.343999999999994,"5min":66.343999999999994,"15min":66.343999999999994},"last":1.0009999999999999}]},{"osd":12,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1679999999999999}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]},{"osd":14,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":3.7749999999999999,"5min":3.7749999999999999,"15min":3.7749999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":40.823,"5min":40.823,"15min":40.823},"last":1.417},{"interface":"front","average":{"1min":5.9349999999999996,"5min":5.9349999999999996,"15min":5.9349999999999996},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":47.441000000000003,"5min":47.441000000000003,"15min":47.441000000000003},"last":1.083}]},{"osd":15,"last update":"Wed Nov 1 04:16:59 2023","interfaces":[{"interface":"back","average":{"1min":11.557,"5min":11.557,"15min":11.557},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":75.683000000000007,"5min":75.683000000000007,"15min":75.683000000000007},"last":1.296},{"interface":"front","average":{"1min":3.806,"5min":3.806,"15min":3.806},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":40.557000000000002,"5min":40.557000000000002,"15min":40.557000000000002},"last":1.51}]}]},{"osd":12,"up_from":22,"seq":94489280525,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":5292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":5419008,"data_stored":6245304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.788}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]},{"osd":13,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":14,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":15,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20430848,"data_stored":35823616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24547328,"data_stored":38391808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12447744,"data_stored":20758528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27119616,"data_stored":42176512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":26054656,"data_stored":43347968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6983680,"data_stored":8769536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":19537920,"data_stored":28790784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8769536,"data_stored":14860288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":17289216,"data_stored":30441472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":12353536,"data_stored":18448384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":12206080,"data_stored":17661952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":19877888,"data_stored":31215616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4567040,"data_stored":5779456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":16019456,"data_stored":29208576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":14942208,"data_stored":22126592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":5980160,"data_stored":7753728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-01T04:17:24.693 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2023-11-01T04:17:24.693 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2023-11-01T04:17:24.742 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:17:24.740+0000 7f07e1573640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:17:24.775 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:17:24.775 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2023-11-01T04:17:24.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2023-11-01T04:17:24.776 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2023-11-01T04:17:24.776 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.776 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:186 3.es0 3:723efae4:::smithi19039307-36:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:24.776 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.963910+0000", 2023-11-01T04:17:24.776 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 67.810690066000006, 2023-11-01T04:17:24.777 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.410473274, 2023-11-01T04:17:24.777 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.777 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:24.777 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.777 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.777 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.778 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 186 2023-11-01T04:17:24.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.778 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963910+0000", 2023-11-01T04:17:24.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963910+0000", 2023-11-01T04:17:24.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.781 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963911+0000", 2023-11-01T04:17:24.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.28e-06 2023-11-01T04:17:24.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.781 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.781 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.782 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963917+0000", 2023-11-01T04:17:24.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1829999999999999e-06 2023-11-01T04:17:24.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.782 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963942+0000", 2023-11-01T04:17:24.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4808e-05 2023-11-01T04:17:24.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.783 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.783 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.963957+0000", 2023-11-01T04:17:24.784 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4698999999999999e-05 2023-11-01T04:17:24.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.967838+0000", 2023-11-01T04:17:24.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003881139 2023-11-01T04:17:24.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.968035+0000", 2023-11-01T04:17:24.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019704400000000001 2023-11-01T04:17:24.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.786 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.786 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:24.786 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.968889+0000", 2023-11-01T04:17:24.786 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00085385099999999998 2023-11-01T04:17:24.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.787 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:24.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.072489+0000", 2023-11-01T04:17:24.787 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.103600028 2023-11-01T04:17:24.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:24.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374278+0000", 2023-11-01T04:17:24.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.301788789 2023-11-01T04:17:24.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374383+0000", 2023-11-01T04:17:24.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010545299999999999 2023-11-01T04:17:24.789 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.789 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.790 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.790 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:187 3.es0 3:723efae4:::smithi19039307-36:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:24.790 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.964019+0000", 2023-11-01T04:17:24.791 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 67.810581303000006, 2023-11-01T04:17:24.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41093530900000003, 2023-11-01T04:17:24.791 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.791 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:24.791 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.792 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.792 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.792 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 187 2023-11-01T04:17:24.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.792 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964019+0000", 2023-11-01T04:17:24.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964019+0000", 2023-11-01T04:17:24.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964020+0000", 2023-11-01T04:17:24.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.26e-06 2023-11-01T04:17:24.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964026+0000", 2023-11-01T04:17:24.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3260000000000001e-06 2023-11-01T04:17:24.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964049+0000", 2023-11-01T04:17:24.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2255e-05 2023-11-01T04:17:24.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.797 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.797 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.964067+0000", 2023-11-01T04:17:24.797 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8643000000000001e-05 2023-11-01T04:17:24.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.969573+0000", 2023-11-01T04:17:24.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055055729999999997 2023-11-01T04:17:24.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.969770+0000", 2023-11-01T04:17:24.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000196787 2023-11-01T04:17:24.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:24.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.971048+0000", 2023-11-01T04:17:24.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012779709999999999 2023-11-01T04:17:24.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:24.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.072491+0000", 2023-11-01T04:17:24.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101443751 2023-11-01T04:17:24.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.801 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:24.802 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374869+0000", 2023-11-01T04:17:24.802 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.302377644 2023-11-01T04:17:24.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.802 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.802 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.374954+0000", 2023-11-01T04:17:24.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5099000000000006e-05 2023-11-01T04:17:24.803 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.803 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.803 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.804 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:194 3.es0 3:706dddd5:::smithi19039307-14:head [copy-from ver 6] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:24.804 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966030+0000", 2023-11-01T04:17:24.804 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 67.808569770000005, 2023-11-01T04:17:24.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.72279825099999995, 2023-11-01T04:17:24.804 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.805 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:24.805 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.805 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.805 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.805 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 194 2023-11-01T04:17:24.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.806 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.806 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966030+0000", 2023-11-01T04:17:24.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.807 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.807 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.807 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.807 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966030+0000", 2023-11-01T04:17:24.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.808 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966032+0000", 2023-11-01T04:17:24.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9470000000000002e-06 2023-11-01T04:17:24.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966043+0000", 2023-11-01T04:17:24.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0396e-05 2023-11-01T04:17:24.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966073+0000", 2023-11-01T04:17:24.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0518000000000002e-05 2023-11-01T04:17:24.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966093+0000", 2023-11-01T04:17:24.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9955000000000001e-05 2023-11-01T04:17:24.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.071960+0000", 2023-11-01T04:17:24.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.105867171 2023-11-01T04:17:24.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.072114+0000", 2023-11-01T04:17:24.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000153323 2023-11-01T04:17:24.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:24.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.480839+0000", 2023-11-01T04:17:24.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40872525500000001 2023-11-01T04:17:24.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:24.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.487356+0000", 2023-11-01T04:17:24.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0065175770000000001 2023-11-01T04:17:24.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:24.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.688636+0000", 2023-11-01T04:17:24.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20127930899999999 2023-11-01T04:17:24.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.688829+0000", 2023-11-01T04:17:24.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0001928 2023-11-01T04:17:24.817 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.817 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.817 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.818 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:195 3.es0 3:706dddd5:::smithi19039307-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:24.818 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966129+0000", 2023-11-01T04:17:24.818 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 67.808471479999994, 2023-11-01T04:17:24.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51772273199999996, 2023-11-01T04:17:24.819 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.819 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:24.819 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.819 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.819 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.819 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 195 2023-11-01T04:17:24.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.820 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966129+0000", 2023-11-01T04:17:24.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966129+0000", 2023-11-01T04:17:24.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966130+0000", 2023-11-01T04:17:24.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4529999999999999e-06 2023-11-01T04:17:24.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966168+0000", 2023-11-01T04:17:24.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7792e-05 2023-11-01T04:17:24.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966192+0000", 2023-11-01T04:17:24.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4083999999999998e-05 2023-11-01T04:17:24.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966210+0000", 2023-11-01T04:17:24.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8196e-05 2023-11-01T04:17:24.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.173008+0000", 2023-11-01T04:17:24.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20679805200000001 2023-11-01T04:17:24.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2023-11-01T04:17:24.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.173168+0000", 2023-11-01T04:17:24.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000159457 2023-11-01T04:17:24.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.483564+0000", 2023-11-01T04:17:24.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.31039652000000001 2023-11-01T04:17:24.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.483711+0000", 2023-11-01T04:17:24.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014698099999999999 2023-11-01T04:17:24.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.483851+0000", 2023-11-01T04:17:24.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000140197 2023-11-01T04:17:24.830 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.830 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.830 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.831 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:202 3.es0 3:7f8a06cd:::smithi19039307-49:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:24.831 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966420+0000", 2023-11-01T04:17:24.831 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 67.808180171999993, 2023-11-01T04:17:24.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.91933000499999995, 2023-11-01T04:17:24.831 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.832 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:24.832 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.832 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.832 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.832 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 202 2023-11-01T04:17:24.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.833 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966420+0000", 2023-11-01T04:17:24.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966420+0000", 2023-11-01T04:17:24.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966421+0000", 2023-11-01T04:17:24.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6199999999999999e-06 2023-11-01T04:17:24.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966431+0000", 2023-11-01T04:17:24.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4660000000000005e-06 2023-11-01T04:17:24.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966456+0000", 2023-11-01T04:17:24.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4748000000000001e-05 2023-11-01T04:17:24.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.837 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966473+0000", 2023-11-01T04:17:24.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7323e-05 2023-11-01T04:17:24.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.375871+0000", 2023-11-01T04:17:24.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40939844800000003 2023-11-01T04:17:24.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.376008+0000", 2023-11-01T04:17:24.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000136997 2023-11-01T04:17:24.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:24.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.585026+0000", 2023-11-01T04:17:24.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.209017436 2023-11-01T04:17:24.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.841 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.841 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:24.841 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.688063+0000", 2023-11-01T04:17:24.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.103037031 2023-11-01T04:17:24.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:24.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.885528+0000", 2023-11-01T04:17:24.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.19746507999999999 2023-11-01T04:17:24.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.843 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.843 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.843 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.885750+0000", 2023-11-01T04:17:24.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022185600000000001 2023-11-01T04:17:24.843 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.844 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.844 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.844 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:203 3.es0 3:7f8a06cd:::smithi19039307-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2023-11-01T04:17:24.844 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:16.966500+0000", 2023-11-01T04:17:24.845 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 67.808099773999999, 2023-11-01T04:17:24.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.62108323700000001, 2023-11-01T04:17:24.845 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.845 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:24.845 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.846 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.846 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.846 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 203 2023-11-01T04:17:24.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.846 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.847 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966500+0000", 2023-11-01T04:17:24.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.847 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966500+0000", 2023-11-01T04:17:24.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.848 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.848 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966501+0000", 2023-11-01T04:17:24.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1570000000000001e-06 2023-11-01T04:17:24.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966530+0000", 2023-11-01T04:17:24.850 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8337999999999999e-05 2023-11-01T04:17:24.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.850 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.850 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966549+0000", 2023-11-01T04:17:24.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9624999999999999e-05 2023-11-01T04:17:24.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:16.966584+0000", 2023-11-01T04:17:24.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4442e-05 2023-11-01T04:17:24.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.852 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.852 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.376334+0000", 2023-11-01T04:17:24.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.409750269 2023-11-01T04:17:24.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2023-11-01T04:17:24.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.376468+0000", 2023-11-01T04:17:24.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000133588 2023-11-01T04:17:24.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.586951+0000", 2023-11-01T04:17:24.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.21048317499999999 2023-11-01T04:17:24.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.587248+0000", 2023-11-01T04:17:24.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029743400000000001 2023-11-01T04:17:24.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:17.587583+0000", 2023-11-01T04:17:24.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000335209 2023-11-01T04:17:24.856 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.856 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.856 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.857 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:272 3.es0 3:706dddd5:::smithi19039307-14:head [read 0~638976,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2023-11-01T04:17:24.857 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:21.017583+0000", 2023-11-01T04:17:24.857 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 63.757017253000001, 2023-11-01T04:17:24.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.412902926, 2023-11-01T04:17:24.857 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.858 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:24.858 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.858 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.858 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.858 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 272 2023-11-01T04:17:24.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.859 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017583+0000", 2023-11-01T04:17:24.860 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.860 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017583+0000", 2023-11-01T04:17:24.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017584+0000", 2023-11-01T04:17:24.862 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6000000000000002e-07 2023-11-01T04:17:24.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017588+0000", 2023-11-01T04:17:24.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1330000000000001e-06 2023-11-01T04:17:24.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017598+0000", 2023-11-01T04:17:24.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8430000000000005e-06 2023-11-01T04:17:24.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017605+0000", 2023-11-01T04:17:24.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7030000000000006e-06 2023-11-01T04:17:24.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017822+0000", 2023-11-01T04:17:24.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000217137 2023-11-01T04:17:24.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2023-11-01T04:17:24.866 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.017832+0000", 2023-11-01T04:17:24.866 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2459999999999992e-06 2023-11-01T04:17:24.866 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.866 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.866 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.226201+0000", 2023-11-01T04:17:24.867 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20836893100000001 2023-11-01T04:17:24.867 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.867 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.867 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.867 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.226369+0000", 2023-11-01T04:17:24.868 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016842100000000001 2023-11-01T04:17:24.868 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.868 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.868 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.868 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:21.430486+0000", 2023-11-01T04:17:24.869 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20411665200000001 2023-11-01T04:17:24.869 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.869 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.869 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.869 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.870 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.870 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:509 3.es0 3:7f8a06cd:::smithi19039307-49:head [read 0~1114112,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2023-11-01T04:17:24.870 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:44.310053+0000", 2023-11-01T04:17:24.870 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 40.464546730999999, 2023-11-01T04:17:24.870 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41447947000000002, 2023-11-01T04:17:24.871 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.871 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:24.871 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.871 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.871 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.871 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 509 2023-11-01T04:17:24.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.872 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.310053+0000", 2023-11-01T04:17:24.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.310053+0000", 2023-11-01T04:17:24.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.310054+0000", 2023-11-01T04:17:24.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7999999999999993e-07 2023-11-01T04:17:24.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.310058+0000", 2023-11-01T04:17:24.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.019e-06 2023-11-01T04:17:24.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.310067+0000", 2023-11-01T04:17:24.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1730000000000001e-06 2023-11-01T04:17:24.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.876 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.410163+0000", 2023-11-01T04:17:24.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100095695 2023-11-01T04:17:24.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.620051+0000", 2023-11-01T04:17:24.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20988817600000001 2023-11-01T04:17:24.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.620336+0000", 2023-11-01T04:17:24.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028468100000000001 2023-11-01T04:17:24.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:44.724533+0000", 2023-11-01T04:17:24.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10419674600000001 2023-11-01T04:17:24.880 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.880 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.880 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.880 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.880 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.881 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:562 3.es0 3:706dddd5:::smithi19039307-14:head [append 0~376832] snapc 22=[22,1e,1c,1b,19,18,12,11,4] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2023-11-01T04:17:24.881 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:49.512179+0000", 2023-11-01T04:17:24.881 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 35.262420544000001, 2023-11-01T04:17:24.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.50653637399999996, 2023-11-01T04:17:24.881 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.881 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:24.882 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.882 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.882 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.882 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 562 2023-11-01T04:17:24.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.883 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.512179+0000", 2023-11-01T04:17:24.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.884 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.884 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.884 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.884 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.512179+0000", 2023-11-01T04:17:24.884 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.885 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.885 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.885 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.512181+0000", 2023-11-01T04:17:24.885 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.141e-06 2023-11-01T04:17:24.885 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.886 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.886 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.886 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.512434+0000", 2023-11-01T04:17:24.886 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025329699999999998 2023-11-01T04:17:24.886 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.887 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.887 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.887 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.512452+0000", 2023-11-01T04:17:24.887 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7660999999999999e-05 2023-11-01T04:17:24.887 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.888 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.888 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.888 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.612562+0000", 2023-11-01T04:17:24.888 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100109973 2023-11-01T04:17:24.888 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.889 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.889 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.613806+0000", 2023-11-01T04:17:24.889 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001244715 2023-11-01T04:17:24.889 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.889 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.890 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.890 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.613974+0000", 2023-11-01T04:17:24.890 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000168185 2023-11-01T04:17:24.890 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.890 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.891 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:24.891 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.615155+0000", 2023-11-01T04:17:24.891 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001180808 2023-11-01T04:17:24.891 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.891 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:24.892 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.616547+0000", 2023-11-01T04:17:24.892 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013913580000000001 2023-11-01T04:17:24.892 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.892 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.892 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:24.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:50.018579+0000", 2023-11-01T04:17:24.893 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.402032426 2023-11-01T04:17:24.893 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.893 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.893 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.893 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:50.018716+0000", 2023-11-01T04:17:24.894 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013681000000000001 2023-11-01T04:17:24.894 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.894 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.894 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.894 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.895 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.895 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:564 3.es0 3:706dddd5:::smithi19039307-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2023-11-01T04:17:24.895 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:49.712722+0000", 2023-11-01T04:17:24.895 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 35.061877758999998, 2023-11-01T04:17:24.895 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51060609800000001, 2023-11-01T04:17:24.895 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.896 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:24.896 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.896 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.896 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.896 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 564 2023-11-01T04:17:24.896 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.897 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.897 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.897 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.897 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.712722+0000", 2023-11-01T04:17:24.897 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.897 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.898 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.898 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.898 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.712722+0000", 2023-11-01T04:17:24.898 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.898 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.899 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.899 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.712724+0000", 2023-11-01T04:17:24.899 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2359999999999999e-06 2023-11-01T04:17:24.899 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.899 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.900 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.712728+0000", 2023-11-01T04:17:24.900 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7600000000000002e-06 2023-11-01T04:17:24.900 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.900 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.900 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.712739+0000", 2023-11-01T04:17:24.901 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1141000000000001e-05 2023-11-01T04:17:24.901 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.901 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.901 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.901 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.712746+0000", 2023-11-01T04:17:24.902 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3450000000000002e-06 2023-11-01T04:17:24.902 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.902 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.902 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.902 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.714070+0000", 2023-11-01T04:17:24.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013245749999999999 2023-11-01T04:17:24.903 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.903 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.903 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:24.903 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:49.714198+0000", 2023-11-01T04:17:24.903 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012794299999999999 2023-11-01T04:17:24.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.904 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.904 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.904 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:50.019194+0000", 2023-11-01T04:17:24.904 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.304995813 2023-11-01T04:17:24.904 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.905 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.905 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:50.019362+0000", 2023-11-01T04:17:24.905 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000167448 2023-11-01T04:17:24.905 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.905 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.906 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.906 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:50.223328+0000", 2023-11-01T04:17:24.906 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20396683700000001 2023-11-01T04:17:24.906 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.906 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.906 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.907 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.907 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.907 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:706 3.es0 3:7f8a06cd:::smithi19039307-49:head [append 0~737280] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:24.907 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:03.445068+0000", 2023-11-01T04:17:24.907 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.329532317999998, 2023-11-01T04:17:24.907 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.72314444700000002, 2023-11-01T04:17:24.908 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.908 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:24.908 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.908 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.908 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.908 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 706 2023-11-01T04:17:24.909 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.909 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.909 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.909 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.909 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.445068+0000", 2023-11-01T04:17:24.909 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.910 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.910 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.910 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.910 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.445068+0000", 2023-11-01T04:17:24.910 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.911 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.911 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.911 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.911 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.445070+0000", 2023-11-01T04:17:24.911 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.74e-06 2023-11-01T04:17:24.911 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.912 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.912 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.445924+0000", 2023-11-01T04:17:24.912 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00085342600000000001 2023-11-01T04:17:24.912 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.912 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.913 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.913 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.445955+0000", 2023-11-01T04:17:24.913 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1304999999999999e-05 2023-11-01T04:17:24.913 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.913 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.913 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.914 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.546060+0000", 2023-11-01T04:17:24.914 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10010466899999999 2023-11-01T04:17:24.914 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.914 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.914 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.546907+0000", 2023-11-01T04:17:24.915 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000846732 2023-11-01T04:17:24.915 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.915 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.915 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.915 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.547096+0000", 2023-11-01T04:17:24.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018942000000000001 2023-11-01T04:17:24.916 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.916 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.916 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:24.916 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.549255+0000", 2023-11-01T04:17:24.916 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021594729999999999 2023-11-01T04:17:24.917 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.917 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.917 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:24.917 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.654676+0000", 2023-11-01T04:17:24.917 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.105420232 2023-11-01T04:17:24.917 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.918 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.918 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:24.918 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.167438+0000", 2023-11-01T04:17:24.918 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.512762261 2023-11-01T04:17:24.918 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.919 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.919 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.919 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.168212+0000", 2023-11-01T04:17:24.919 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000774189 2023-11-01T04:17:24.919 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.919 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.920 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.920 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.920 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.920 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:707 3.es0 3:7f8a06cd:::smithi19039307-49:head [append 0~557056] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:24.920 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:03.546123+0000", 2023-11-01T04:17:24.920 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.228476918999998, 2023-11-01T04:17:24.921 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.62322876299999996, 2023-11-01T04:17:24.921 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.921 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:24.921 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.921 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.922 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.922 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 707 2023-11-01T04:17:24.922 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.922 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.922 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.922 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.923 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.546123+0000", 2023-11-01T04:17:24.923 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.923 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.923 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.923 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.924 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.546123+0000", 2023-11-01T04:17:24.924 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.924 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.924 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.924 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.925 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.546125+0000", 2023-11-01T04:17:24.925 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2900000000000001e-06 2023-11-01T04:17:24.925 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.925 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.926 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.546773+0000", 2023-11-01T04:17:24.926 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064774400000000001 2023-11-01T04:17:24.926 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.926 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.926 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.927 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.546800+0000", 2023-11-01T04:17:24.927 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6579000000000001e-05 2023-11-01T04:17:24.927 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.927 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.928 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.546819+0000", 2023-11-01T04:17:24.928 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9448000000000001e-05 2023-11-01T04:17:24.928 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.928 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.928 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.609012+0000", 2023-11-01T04:17:24.929 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.062192781000000003 2023-11-01T04:17:24.929 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.929 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.929 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.929 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.609201+0000", 2023-11-01T04:17:24.930 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018883600000000001 2023-11-01T04:17:24.930 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.930 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.930 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:24.930 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.610704+0000", 2023-11-01T04:17:24.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015032660000000001 2023-11-01T04:17:24.931 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.931 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.931 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:24.931 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.654682+0000", 2023-11-01T04:17:24.931 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.043978072999999999 2023-11-01T04:17:24.932 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.932 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.932 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:24.932 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.168608+0000", 2023-11-01T04:17:24.932 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51392625300000006 2023-11-01T04:17:24.933 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.933 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.933 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.933 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.169352+0000", 2023-11-01T04:17:24.933 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00074349300000000002 2023-11-01T04:17:24.933 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.934 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.934 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.934 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.934 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.934 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:708 3.es0 3:7f8a06cd:::smithi19039307-49:head [append 0~606208] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:24.934 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:03.546858+0000", 2023-11-01T04:17:24.935 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.227742073000002, 2023-11-01T04:17:24.935 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.67282281200000005, 2023-11-01T04:17:24.935 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.935 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:24.935 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.935 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.936 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.936 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 708 2023-11-01T04:17:24.936 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.936 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.936 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.936 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.546858+0000", 2023-11-01T04:17:24.937 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.937 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.937 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.937 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.937 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.546858+0000", 2023-11-01T04:17:24.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.938 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.938 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.938 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.938 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.546859+0000", 2023-11-01T04:17:24.938 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.403e-06 2023-11-01T04:17:24.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.939 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.939 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.939 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.547725+0000", 2023-11-01T04:17:24.939 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00086536200000000005 2023-11-01T04:17:24.939 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.940 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.940 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.940 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.547750+0000", 2023-11-01T04:17:24.940 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4802000000000001e-05 2023-11-01T04:17:24.940 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.941 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.941 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.547767+0000", 2023-11-01T04:17:24.941 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7365999999999999e-05 2023-11-01T04:17:24.941 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.941 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.942 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.942 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.612011+0000", 2023-11-01T04:17:24.942 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.064243876000000005 2023-11-01T04:17:24.942 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.942 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.942 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.612325+0000", 2023-11-01T04:17:24.943 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031440299999999999 2023-11-01T04:17:24.943 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.943 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.943 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:24.943 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.614304+0000", 2023-11-01T04:17:24.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019789819999999998 2023-11-01T04:17:24.944 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.944 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.944 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:24.944 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.654686+0000", 2023-11-01T04:17:24.944 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.040381470000000003 2023-11-01T04:17:24.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.945 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.945 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:24.945 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.218923+0000", 2023-11-01T04:17:24.945 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.56423753499999996 2023-11-01T04:17:24.945 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.946 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.946 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.946 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.219681+0000", 2023-11-01T04:17:24.946 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00075761299999999997 2023-11-01T04:17:24.946 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.946 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.947 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.947 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.947 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.947 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:709 3.es0 3:7f8a06cd:::smithi19039307-49:head [setxattr _header (57) in=7b] snapc 2f=[2f,28,22,19] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:24.947 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:03.547795+0000", 2023-11-01T04:17:24.947 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.226804725000001, 2023-11-01T04:17:24.948 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.67262054400000004, 2023-11-01T04:17:24.948 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.948 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:24.948 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.948 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.948 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.949 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 709 2023-11-01T04:17:24.949 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.949 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.949 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.949 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.949 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.547795+0000", 2023-11-01T04:17:24.950 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.950 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.950 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.950 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.950 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.547795+0000", 2023-11-01T04:17:24.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.951 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.951 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.951 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.951 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.547797+0000", 2023-11-01T04:17:24.951 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7400000000000001e-06 2023-11-01T04:17:24.952 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.952 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.952 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.952 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.547805+0000", 2023-11-01T04:17:24.952 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6860000000000006e-06 2023-11-01T04:17:24.953 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.953 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.953 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.953 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.547825+0000", 2023-11-01T04:17:24.953 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0219000000000001e-05 2023-11-01T04:17:24.953 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.954 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.954 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.547840+0000", 2023-11-01T04:17:24.954 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4762e-05 2023-11-01T04:17:24.954 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.954 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.955 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.955 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.651723+0000", 2023-11-01T04:17:24.955 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.103883448 2023-11-01T04:17:24.955 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.955 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.956 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.956 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.652063+0000", 2023-11-01T04:17:24.956 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033982400000000002 2023-11-01T04:17:24.956 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.956 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.956 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:24.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.653529+0000", 2023-11-01T04:17:24.957 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014656420000000001 2023-11-01T04:17:24.957 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.957 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.957 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:24.957 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.655417+0000", 2023-11-01T04:17:24.958 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018884010000000001 2023-11-01T04:17:24.958 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.958 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.958 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:24.958 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.219983+0000", 2023-11-01T04:17:24.959 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.56456637899999995 2023-11-01T04:17:24.959 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.959 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.959 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.959 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.220416+0000", 2023-11-01T04:17:24.960 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043244300000000002 2023-11-01T04:17:24.960 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.960 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.960 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.960 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.960 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.961 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:710 3.es0 3:7f8a06cd:::smithi19039307-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2023-11-01T04:17:24.961 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:03.547859+0000", 2023-11-01T04:17:24.961 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.226741411999999, 2023-11-01T04:17:24.961 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0769060319999999, 2023-11-01T04:17:24.961 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.962 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:24.962 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.962 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.962 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.962 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 710 2023-11-01T04:17:24.963 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.963 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.963 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.963 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.963 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.547859+0000", 2023-11-01T04:17:24.963 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.964 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.964 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.964 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.964 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.547859+0000", 2023-11-01T04:17:24.964 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.965 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.965 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.965 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.965 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.547860+0000", 2023-11-01T04:17:24.965 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2500000000000001e-06 2023-11-01T04:17:24.966 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.966 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.966 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.966 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.547865+0000", 2023-11-01T04:17:24.966 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1059999999999999e-06 2023-11-01T04:17:24.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.967 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.967 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.967 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.547884+0000", 2023-11-01T04:17:24.967 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8749e-05 2023-11-01T04:17:24.967 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.968 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.968 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.547899+0000", 2023-11-01T04:17:24.968 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5265999999999999e-05 2023-11-01T04:17:24.968 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.968 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.969 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.654369+0000", 2023-11-01T04:17:24.969 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.106470467 2023-11-01T04:17:24.969 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.969 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.969 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for rw locks", 2023-11-01T04:17:24.970 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.654658+0000", 2023-11-01T04:17:24.970 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028880399999999998 2023-11-01T04:17:24.970 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.970 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.970 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.233425+0000", 2023-11-01T04:17:24.971 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.57876637099999995 2023-11-01T04:17:24.971 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.971 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.971 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.971 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.233612+0000", 2023-11-01T04:17:24.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018784300000000001 2023-11-01T04:17:24.972 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.972 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.972 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.972 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:04.624765+0000", 2023-11-01T04:17:24.972 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.39115217600000002 2023-11-01T04:17:24.973 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.973 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.973 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.973 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.973 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.974 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:846 3.es0 3:7f8a06cd:::smithi19039307-49:head [stat,delete] snapc 39=[39,38,37,35,34,2f,28,19] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2023-11-01T04:17:24.974 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:14.541437+0000", 2023-11-01T04:17:24.974 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 10.233163470999999, 2023-11-01T04:17:24.974 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.47729008099999998, 2023-11-01T04:17:24.974 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.974 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2023-11-01T04:17:24.975 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.975 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.975 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.975 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 846 2023-11-01T04:17:24.975 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.975 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.976 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.976 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:14.541437+0000", 2023-11-01T04:17:24.976 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.976 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.976 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.977 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:14.541437+0000", 2023-11-01T04:17:24.977 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.977 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.977 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.977 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.978 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:14.541439+0000", 2023-11-01T04:17:24.978 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8399999999999999e-06 2023-11-01T04:17:24.978 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.978 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.978 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:14.541451+0000", 2023-11-01T04:17:24.979 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1673e-05 2023-11-01T04:17:24.979 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.979 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.979 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.979 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:14.541484+0000", 2023-11-01T04:17:24.980 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2712000000000001e-05 2023-11-01T04:17:24.980 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.980 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.980 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.980 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:14.541500+0000", 2023-11-01T04:17:24.981 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5829e-05 2023-11-01T04:17:24.981 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.981 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.981 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.981 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:14.741866+0000", 2023-11-01T04:17:24.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20036627500000001 2023-11-01T04:17:24.982 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.982 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.982 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.982 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:14.742074+0000", 2023-11-01T04:17:24.982 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020857100000000001 2023-11-01T04:17:24.983 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.983 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.983 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.983 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:15.018727+0000", 2023-11-01T04:17:24.983 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.276652181 2023-11-01T04:17:24.983 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.984 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.984 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.984 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.984 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.984 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:964 3.es0 3:7c69a271:::smithi19039307-13:head [append 0~802816] snapc 3f=[3f,3e,3d,3a,39,37,35,34,28,19] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-11-01T04:17:24.984 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:21.145335+0000", 2023-11-01T04:17:24.985 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 3.6292645650000002, 2023-11-01T04:17:24.985 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.56489595699999995, 2023-11-01T04:17:24.985 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:24.985 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:24.985 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:24.986 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:24.986 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:24.986 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 964 2023-11-01T04:17:24.986 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.986 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:24.986 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:24.987 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.145335+0000", 2023-11-01T04:17:24.987 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.987 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.987 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.987 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:24.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.145335+0000", 2023-11-01T04:17:24.988 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:24.988 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.988 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.988 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:24.988 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.145339+0000", 2023-11-01T04:17:24.989 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5389999999999999e-06 2023-11-01T04:17:24.989 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.989 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.989 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:24.989 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.146488+0000", 2023-11-01T04:17:24.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001149024 2023-11-01T04:17:24.990 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.990 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.990 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:24.990 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.146524+0000", 2023-11-01T04:17:24.990 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6139000000000001e-05 2023-11-01T04:17:24.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.991 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.991 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:24.991 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.146543+0000", 2023-11-01T04:17:24.991 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8555000000000001e-05 2023-11-01T04:17:24.991 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.992 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.992 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.146742+0000", 2023-11-01T04:17:24.992 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019966299999999999 2023-11-01T04:17:24.992 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.992 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2023-11-01T04:17:24.993 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.146773+0000", 2023-11-01T04:17:24.993 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0414999999999999e-05 2023-11-01T04:17:24.993 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.993 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.993 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:24.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.152334+0000", 2023-11-01T04:17:24.994 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055614849999999997 2023-11-01T04:17:24.994 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.994 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.994 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:24.994 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.152612+0000", 2023-11-01T04:17:24.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027816100000000002 2023-11-01T04:17:24.995 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.995 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:24.995 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.155547+0000", 2023-11-01T04:17:24.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0029344919999999999 2023-11-01T04:17:24.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:24.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.161927+0000", 2023-11-01T04:17:24.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063801079999999998 2023-11-01T04:17:24.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:24.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.709361+0000", 2023-11-01T04:17:24.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.54743370300000005 2023-11-01T04:17:24.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:24.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:24.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.710231+0000", 2023-11-01T04:17:24.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00087067299999999998 2023-11-01T04:17:24.999 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.999 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:24.999 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:24.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:24.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.000 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:965 3.es0 3:7c69a271:::smithi19039307-13:head [append 0~425984] snapc 3f=[3f,3e,3d,3a,39,37,35,34,28,19] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-11-01T04:17:25.000 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:21.151119+0000", 2023-11-01T04:17:25.000 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 3.6234810770000001, 2023-11-01T04:17:25.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.56035065900000003, 2023-11-01T04:17:25.000 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.001 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.001 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:25.001 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:25.001 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:25.001 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 965 2023-11-01T04:17:25.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.002 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.151119+0000", 2023-11-01T04:17:25.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.151119+0000", 2023-11-01T04:17:25.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.151122+0000", 2023-11-01T04:17:25.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.847e-06 2023-11-01T04:17:25.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.151591+0000", 2023-11-01T04:17:25.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00046902700000000001 2023-11-01T04:17:25.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.151617+0000", 2023-11-01T04:17:25.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6035000000000001e-05 2023-11-01T04:17:25.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.151632+0000", 2023-11-01T04:17:25.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5359999999999999e-05 2023-11-01T04:17:25.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.157703+0000", 2023-11-01T04:17:25.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006071206 2023-11-01T04:17:25.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.158030+0000", 2023-11-01T04:17:25.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00032629600000000001 2023-11-01T04:17:25.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:25.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.160026+0000", 2023-11-01T04:17:25.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019966599999999999 2023-11-01T04:17:25.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:25.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.162615+0000", 2023-11-01T04:17:25.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025888959999999998 2023-11-01T04:17:25.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.710662+0000", 2023-11-01T04:17:25.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.54804650700000002 2023-11-01T04:17:25.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.711470+0000", 2023-11-01T04:17:25.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00080782499999999997 2023-11-01T04:17:25.012 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.012 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.012 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.013 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:966 3.es0 3:7c69a271:::smithi19039307-13:head [append 0~720896] snapc 3f=[3f,3e,3d,3a,39,37,35,34,28,19] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-11-01T04:17:25.013 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:21.160390+0000", 2023-11-01T04:17:25.013 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 3.614209792, 2023-11-01T04:17:25.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.55202144500000006, 2023-11-01T04:17:25.014 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.014 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.014 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:25.014 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:25.014 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:25.014 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 966 2023-11-01T04:17:25.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.015 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.160390+0000", 2023-11-01T04:17:25.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.160390+0000", 2023-11-01T04:17:25.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.160393+0000", 2023-11-01T04:17:25.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9869999999999999e-06 2023-11-01T04:17:25.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.161474+0000", 2023-11-01T04:17:25.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001080765 2023-11-01T04:17:25.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.161513+0000", 2023-11-01T04:17:25.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9338999999999997e-05 2023-11-01T04:17:25.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.161536+0000", 2023-11-01T04:17:25.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2789000000000002e-05 2023-11-01T04:17:25.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.162830+0000", 2023-11-01T04:17:25.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001293513 2023-11-01T04:17:25.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.162917+0000", 2023-11-01T04:17:25.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7822e-05 2023-11-01T04:17:25.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:25.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.163768+0000", 2023-11-01T04:17:25.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00085084899999999998 2023-11-01T04:17:25.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:25.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.165102+0000", 2023-11-01T04:17:25.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013341259999999999 2023-11-01T04:17:25.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.711938+0000", 2023-11-01T04:17:25.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.54683597799999994 2023-11-01T04:17:25.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.712412+0000", 2023-11-01T04:17:25.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047327699999999998 2023-11-01T04:17:25.025 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.026 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.026 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.026 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.4559.0:967 3.es0 3:7c69a271:::smithi19039307-13:head [append 0~409600] snapc 3f=[3f,3e,3d,3a,39,37,35,34,28,19] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-11-01T04:17:25.026 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:21.164569+0000", 2023-11-01T04:17:25.027 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 3.610030971, 2023-11-01T04:17:25.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.64865365399999997, 2023-11-01T04:17:25.027 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.027 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.027 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2023-11-01T04:17:25.027 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.4559", 2023-11-01T04:17:25.028 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/403659509", 2023-11-01T04:17:25.028 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 967 2023-11-01T04:17:25.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.028 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.164569+0000", 2023-11-01T04:17:25.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.164569+0000", 2023-11-01T04:17:25.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.164573+0000", 2023-11-01T04:17:25.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6270000000000002e-06 2023-11-01T04:17:25.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.165076+0000", 2023-11-01T04:17:25.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000503442 2023-11-01T04:17:25.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.165105+0000", 2023-11-01T04:17:25.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8878999999999999e-05 2023-11-01T04:17:25.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.165121+0000", 2023-11-01T04:17:25.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6365999999999999e-05 2023-11-01T04:17:25.034 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.034 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.034 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.034 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.165214+0000", 2023-11-01T04:17:25.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2285000000000002e-05 2023-11-01T04:17:25.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.165290+0000", 2023-11-01T04:17:25.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6044999999999994e-05 2023-11-01T04:17:25.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:25.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.165889+0000", 2023-11-01T04:17:25.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00059978899999999999 2023-11-01T04:17:25.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:25.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.168660+0000", 2023-11-01T04:17:25.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002770355 2023-11-01T04:17:25.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.812394+0000", 2023-11-01T04:17:25.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.643733742 2023-11-01T04:17:25.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.038 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:21.813223+0000", 2023-11-01T04:17:25.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00082912399999999999 2023-11-01T04:17:25.039 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.039 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.039 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.040 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.040 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.040 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:17:25.041 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2023-11-01T04:17:25.042 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:17:25.038+0000 7f83cdfc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:17:25.118 INFO:teuthology.orchestra.run.smithi153.stderr:noscrub is unset 2023-11-01T04:17:25.118 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'unique_pool_0' removed 2023-11-01T04:17:25.142 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:17:25.140+0000 7efe065b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:17:25.147 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2023-11-01T04:17:25.156 INFO:tasks.thrashosds:joining thrashosds 2023-11-01T04:17:25.174 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:17:25.174 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:17:25.174 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2023-11-01T04:17:25.174 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:17:25.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2023-11-01T04:17:25.303 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:17:25.304 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2023-11-01T04:17:25.304 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2023-11-01T04:17:25.304 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2023-11-01T04:17:25.304 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:17:25.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2023-11-01T04:17:25.443 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-11-01T04:17:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2023-11-01T04:17:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2023-11-01T04:17:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2023-11-01T04:17:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:36 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'22, mlcod=25'16)", 2023-11-01T04:17:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:00.688757+0000", 2023-11-01T04:17:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 84.752820134999993, 2023-11-01T04:17:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20189465300000001, 2023-11-01T04:17:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688757+0000", 2023-11-01T04:17:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688757+0000", 2023-11-01T04:17:25.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688758+0000", 2023-11-01T04:17:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5300000000000004e-07 2023-11-01T04:17:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688763+0000", 2023-11-01T04:17:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7389999999999996e-06 2023-11-01T04:17:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688791+0000", 2023-11-01T04:17:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8136999999999998e-05 2023-11-01T04:17:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688799+0000", 2023-11-01T04:17:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3390000000000004e-06 2023-11-01T04:17:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.888843+0000", 2023-11-01T04:17:25.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20004432999999999 2023-11-01T04:17:25.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.888878+0000", 2023-11-01T04:17:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4814000000000002e-05 2023-11-01T04:17:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.890502+0000", 2023-11-01T04:17:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001623813 2023-11-01T04:17:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.890652+0000", 2023-11-01T04:17:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014982799999999999 2023-11-01T04:17:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:37 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'23, mlcod=25'16)", 2023-11-01T04:17:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:00.688817+0000", 2023-11-01T04:17:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 84.752760335000005, 2023-11-01T04:17:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.202682319, 2023-11-01T04:17:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688817+0000", 2023-11-01T04:17:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688817+0000", 2023-11-01T04:17:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688817+0000", 2023-11-01T04:17:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6700000000000003e-07 2023-11-01T04:17:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688823+0000", 2023-11-01T04:17:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1589999999999998e-06 2023-11-01T04:17:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.688835+0000", 2023-11-01T04:17:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1472e-05 2023-11-01T04:17:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.788922+0000", 2023-11-01T04:17:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100087538 2023-11-01T04:17:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.889458+0000", 2023-11-01T04:17:25.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100535469 2023-11-01T04:17:25.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.889515+0000", 2023-11-01T04:17:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6990999999999997e-05 2023-11-01T04:17:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.891416+0000", 2023-11-01T04:17:25.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019012409999999999 2023-11-01T04:17:25.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:00.891499+0000", 2023-11-01T04:17:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2881999999999997e-05 2023-11-01T04:17:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:53 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'31, mlcod=26'29)", 2023-11-01T04:17:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:01.627435+0000", 2023-11-01T04:17:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 83.814142036999996, 2023-11-01T04:17:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20423751000000001, 2023-11-01T04:17:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627435+0000", 2023-11-01T04:17:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627435+0000", 2023-11-01T04:17:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627437+0000", 2023-11-01T04:17:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6670000000000001e-06 2023-11-01T04:17:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627449+0000", 2023-11-01T04:17:25.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2571999999999999e-05 2023-11-01T04:17:25.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627477+0000", 2023-11-01T04:17:25.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.741e-05 2023-11-01T04:17:25.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627491+0000", 2023-11-01T04:17:25.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3953e-05 2023-11-01T04:17:25.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627659+0000", 2023-11-01T04:17:25.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016794299999999999 2023-11-01T04:17:25.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.474 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.627699+0000", 2023-11-01T04:17:25.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0036000000000001e-05 2023-11-01T04:17:25.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831530+0000", 2023-11-01T04:17:25.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20383152299999999 2023-11-01T04:17:25.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831672+0000", 2023-11-01T04:17:25.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014240600000000001 2023-11-01T04:17:25.476 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.476 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.476 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.477 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:54 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'32, mlcod=26'29)", 2023-11-01T04:17:25.477 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:01.628360+0000", 2023-11-01T04:17:25.477 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 83.813217257999995, 2023-11-01T04:17:25.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20340096899999999, 2023-11-01T04:17:25.478 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.478 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.478 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628360+0000", 2023-11-01T04:17:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628360+0000", 2023-11-01T04:17:25.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628362+0000", 2023-11-01T04:17:25.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.787e-06 2023-11-01T04:17:25.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628374+0000", 2023-11-01T04:17:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2499e-05 2023-11-01T04:17:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628399+0000", 2023-11-01T04:17:25.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5037000000000001e-05 2023-11-01T04:17:25.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628412+0000", 2023-11-01T04:17:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2982e-05 2023-11-01T04:17:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628555+0000", 2023-11-01T04:17:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000142783 2023-11-01T04:17:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.628587+0000", 2023-11-01T04:17:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2026999999999997e-05 2023-11-01T04:17:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831677+0000", 2023-11-01T04:17:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20309039200000001 2023-11-01T04:17:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831761+0000", 2023-11-01T04:17:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3461999999999995e-05 2023-11-01T04:17:25.488 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.488 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.488 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:55 1.0 e26/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 26'33, mlcod=26'29)", 2023-11-01T04:17:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:01.629257+0000", 2023-11-01T04:17:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 83.812320266, 2023-11-01T04:17:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20259303100000001, 2023-11-01T04:17:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629257+0000", 2023-11-01T04:17:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629257+0000", 2023-11-01T04:17:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629259+0000", 2023-11-01T04:17:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8530000000000001e-06 2023-11-01T04:17:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629358+0000", 2023-11-01T04:17:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9019999999999997e-05 2023-11-01T04:17:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629386+0000", 2023-11-01T04:17:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8453999999999999e-05 2023-11-01T04:17:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.629401+0000", 2023-11-01T04:17:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4465e-05 2023-11-01T04:17:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.829838+0000", 2023-11-01T04:17:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20043709100000001 2023-11-01T04:17:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.829873+0000", 2023-11-01T04:17:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5253000000000003e-05 2023-11-01T04:17:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831769+0000", 2023-11-01T04:17:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018957100000000001 2023-11-01T04:17:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.499 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.499 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:01.831850+0000", 2023-11-01T04:17:25.499 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1185000000000003e-05 2023-11-01T04:17:25.499 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.499 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:104 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'66, mlcod=28'64)", 2023-11-01T04:17:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.140797+0000", 2023-11-01T04:17:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 80.300779973999994, 2023-11-01T04:17:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20170181600000001, 2023-11-01T04:17:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.140797+0000", 2023-11-01T04:17:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.140797+0000", 2023-11-01T04:17:25.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.140799+0000", 2023-11-01T04:17:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.176e-06 2023-11-01T04:17:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.140813+0000", 2023-11-01T04:17:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3791999999999999e-05 2023-11-01T04:17:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.506 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.140846+0000", 2023-11-01T04:17:25.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2824000000000003e-05 2023-11-01T04:17:25.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.240986+0000", 2023-11-01T04:17:25.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100140625 2023-11-01T04:17:25.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241142+0000", 2023-11-01T04:17:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015555899999999999 2023-11-01T04:17:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241199+0000", 2023-11-01T04:17:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7321999999999998e-05 2023-11-01T04:17:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.342429+0000", 2023-11-01T04:17:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101229401 2023-11-01T04:17:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.342499+0000", 2023-11-01T04:17:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0117000000000006e-05 2023-11-01T04:17:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.512 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:105 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'67, mlcod=28'64)", 2023-11-01T04:17:25.512 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.241023+0000", 2023-11-01T04:17:25.512 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 80.200553537999994, 2023-11-01T04:17:25.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10730198000000001, 2023-11-01T04:17:25.513 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.513 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.513 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241023+0000", 2023-11-01T04:17:25.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241023+0000", 2023-11-01T04:17:25.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241024+0000", 2023-11-01T04:17:25.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9999999999999995e-07 2023-11-01T04:17:25.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241045+0000", 2023-11-01T04:17:25.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0684999999999999e-05 2023-11-01T04:17:25.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241062+0000", 2023-11-01T04:17:25.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6956000000000002e-05 2023-11-01T04:17:25.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.518 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.518 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.518 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241072+0000", 2023-11-01T04:17:25.519 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0329000000000001e-05 2023-11-01T04:17:25.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.519 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.341337+0000", 2023-11-01T04:17:25.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100264753 2023-11-01T04:17:25.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.341371+0000", 2023-11-01T04:17:25.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4241e-05 2023-11-01T04:17:25.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.348245+0000", 2023-11-01T04:17:25.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068737700000000004 2023-11-01T04:17:25.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.348325+0000", 2023-11-01T04:17:25.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0246000000000004e-05 2023-11-01T04:17:25.523 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.523 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.523 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.524 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:106 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'68, mlcod=28'64)", 2023-11-01T04:17:25.524 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.241117+0000", 2023-11-01T04:17:25.524 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 80.200459555999998, 2023-11-01T04:17:25.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10725122099999999, 2023-11-01T04:17:25.525 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.525 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.525 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.525 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.525 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241117+0000", 2023-11-01T04:17:25.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.526 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241117+0000", 2023-11-01T04:17:25.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241118+0000", 2023-11-01T04:17:25.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8400000000000004e-07 2023-11-01T04:17:25.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241174+0000", 2023-11-01T04:17:25.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6097999999999997e-05 2023-11-01T04:17:25.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241190+0000", 2023-11-01T04:17:25.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5545000000000001e-05 2023-11-01T04:17:25.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241201+0000", 2023-11-01T04:17:25.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1148999999999999e-05 2023-11-01T04:17:25.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.341896+0000", 2023-11-01T04:17:25.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100695009 2023-11-01T04:17:25.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.341927+0000", 2023-11-01T04:17:25.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1597000000000003e-05 2023-11-01T04:17:25.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.348328+0000", 2023-11-01T04:17:25.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006401042 2023-11-01T04:17:25.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.348369+0000", 2023-11-01T04:17:25.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0197000000000002e-05 2023-11-01T04:17:25.535 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.535 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.535 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.535 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:107 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'69, mlcod=28'64)", 2023-11-01T04:17:25.536 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.241214+0000", 2023-11-01T04:17:25.536 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 80.200362570999999, 2023-11-01T04:17:25.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.109683828, 2023-11-01T04:17:25.536 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.536 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.536 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.537 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241214+0000", 2023-11-01T04:17:25.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241214+0000", 2023-11-01T04:17:25.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241215+0000", 2023-11-01T04:17:25.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7999999999999995e-07 2023-11-01T04:17:25.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241224+0000", 2023-11-01T04:17:25.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1630000000000002e-06 2023-11-01T04:17:25.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241233+0000", 2023-11-01T04:17:25.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3520000000000007e-06 2023-11-01T04:17:25.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.241239+0000", 2023-11-01T04:17:25.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1890000000000004e-06 2023-11-01T04:17:25.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.342619+0000", 2023-11-01T04:17:25.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101380392 2023-11-01T04:17:25.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.342679+0000", 2023-11-01T04:17:25.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0387999999999998e-05 2023-11-01T04:17:25.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350812+0000", 2023-11-01T04:17:25.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0081325249999999998 2023-11-01T04:17:25.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.546 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350898+0000", 2023-11-01T04:17:25.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6238999999999999e-05 2023-11-01T04:17:25.546 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.547 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.547 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.547 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:108 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'70, mlcod=28'64)", 2023-11-01T04:17:25.547 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.242130+0000", 2023-11-01T04:17:25.548 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 80.199446871999996, 2023-11-01T04:17:25.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108814548, 2023-11-01T04:17:25.548 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.548 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.548 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242130+0000", 2023-11-01T04:17:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242130+0000", 2023-11-01T04:17:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242132+0000", 2023-11-01T04:17:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7799999999999999e-06 2023-11-01T04:17:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242144+0000", 2023-11-01T04:17:25.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2201999999999999e-05 2023-11-01T04:17:25.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242174+0000", 2023-11-01T04:17:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9484000000000001e-05 2023-11-01T04:17:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.242192+0000", 2023-11-01T04:17:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8862e-05 2023-11-01T04:17:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.343513+0000", 2023-11-01T04:17:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101320113 2023-11-01T04:17:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.343569+0000", 2023-11-01T04:17:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6158e-05 2023-11-01T04:17:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350902+0000", 2023-11-01T04:17:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0073331500000000001 2023-11-01T04:17:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350945+0000", 2023-11-01T04:17:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2799000000000001e-05 2023-11-01T04:17:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:109 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'71, mlcod=28'64)", 2023-11-01T04:17:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.243545+0000", 2023-11-01T04:17:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 80.198032280999996, 2023-11-01T04:17:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.107439464, 2023-11-01T04:17:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243545+0000", 2023-11-01T04:17:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243545+0000", 2023-11-01T04:17:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243547+0000", 2023-11-01T04:17:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8530000000000001e-06 2023-11-01T04:17:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243603+0000", 2023-11-01T04:17:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6498e-05 2023-11-01T04:17:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243632+0000", 2023-11-01T04:17:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9135000000000001e-05 2023-11-01T04:17:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.243646+0000", 2023-11-01T04:17:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3845e-05 2023-11-01T04:17:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.344360+0000", 2023-11-01T04:17:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10071392699999999 2023-11-01T04:17:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.344410+0000", 2023-11-01T04:17:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0078999999999999e-05 2023-11-01T04:17:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350947+0000", 2023-11-01T04:17:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0065365129999999999 2023-11-01T04:17:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.350984+0000", 2023-11-01T04:17:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7614000000000003e-05 2023-11-01T04:17:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:110 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'72, mlcod=28'64)", 2023-11-01T04:17:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.244933+0000", 2023-11-01T04:17:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 80.196644313999997, 2023-11-01T04:17:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.107682845, 2023-11-01T04:17:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244933+0000", 2023-11-01T04:17:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244933+0000", 2023-11-01T04:17:25.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244934+0000", 2023-11-01T04:17:25.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.776e-06 2023-11-01T04:17:25.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244947+0000", 2023-11-01T04:17:25.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2046e-05 2023-11-01T04:17:25.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244976+0000", 2023-11-01T04:17:25.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9141000000000001e-05 2023-11-01T04:17:25.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.244989+0000", 2023-11-01T04:17:25.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3675e-05 2023-11-01T04:17:25.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.345397+0000", 2023-11-01T04:17:25.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100407617 2023-11-01T04:17:25.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.584 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.345451+0000", 2023-11-01T04:17:25.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4475999999999998e-05 2023-11-01T04:17:25.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.352527+0000", 2023-11-01T04:17:25.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070755089999999998 2023-11-01T04:17:25.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.352616+0000", 2023-11-01T04:17:25.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8604999999999996e-05 2023-11-01T04:17:25.586 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.587 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.587 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.587 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:118 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'80, mlcod=28'78)", 2023-11-01T04:17:25.587 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.860484+0000", 2023-11-01T04:17:25.588 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 79.581092659999996, 2023-11-01T04:17:25.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20313931199999999, 2023-11-01T04:17:25.588 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.588 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.588 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860484+0000", 2023-11-01T04:17:25.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860484+0000", 2023-11-01T04:17:25.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.590 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860487+0000", 2023-11-01T04:17:25.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5239999999999999e-06 2023-11-01T04:17:25.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860560+0000", 2023-11-01T04:17:25.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3109999999999996e-05 2023-11-01T04:17:25.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860593+0000", 2023-11-01T04:17:25.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3077000000000002e-05 2023-11-01T04:17:25.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.860611+0000", 2023-11-01T04:17:25.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7881999999999999e-05 2023-11-01T04:17:25.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.960884+0000", 2023-11-01T04:17:25.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100273563 2023-11-01T04:17:25.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.960949+0000", 2023-11-01T04:17:25.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4250000000000003e-05 2023-11-01T04:17:25.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063550+0000", 2023-11-01T04:17:25.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.102601553 2023-11-01T04:17:25.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063624+0000", 2023-11-01T04:17:25.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3353000000000001e-05 2023-11-01T04:17:25.597 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.597 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.598 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.598 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:119 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'81, mlcod=28'78)", 2023-11-01T04:17:25.598 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.861924+0000", 2023-11-01T04:17:25.598 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 79.579653484000005, 2023-11-01T04:17:25.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.201754672, 2023-11-01T04:17:25.599 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.599 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.599 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.861924+0000", 2023-11-01T04:17:25.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.600 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.861924+0000", 2023-11-01T04:17:25.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.861925+0000", 2023-11-01T04:17:25.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.663e-06 2023-11-01T04:17:25.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.602 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.862002+0000", 2023-11-01T04:17:25.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7217000000000003e-05 2023-11-01T04:17:25.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.603 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.603 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.862024+0000", 2023-11-01T04:17:25.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1957999999999999e-05 2023-11-01T04:17:25.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.604 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.862035+0000", 2023-11-01T04:17:25.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0482000000000001e-05 2023-11-01T04:17:25.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.961997+0000", 2023-11-01T04:17:25.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.099962336999999998 2023-11-01T04:17:25.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.962032+0000", 2023-11-01T04:17:25.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4804000000000001e-05 2023-11-01T04:17:25.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063634+0000", 2023-11-01T04:17:25.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.101602151 2023-11-01T04:17:25.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.608 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.608 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063678+0000", 2023-11-01T04:17:25.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4060000000000002e-05 2023-11-01T04:17:25.609 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.609 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.609 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.609 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:120 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'82, mlcod=28'78)", 2023-11-01T04:17:25.610 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.863328+0000", 2023-11-01T04:17:25.610 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 79.578249012000001, 2023-11-01T04:17:25.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20039775900000001, 2023-11-01T04:17:25.610 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.610 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.611 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863328+0000", 2023-11-01T04:17:25.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863328+0000", 2023-11-01T04:17:25.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863330+0000", 2023-11-01T04:17:25.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7770000000000001e-06 2023-11-01T04:17:25.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863442+0000", 2023-11-01T04:17:25.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112596 2023-11-01T04:17:25.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863472+0000", 2023-11-01T04:17:25.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9490999999999999e-05 2023-11-01T04:17:25.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.863486+0000", 2023-11-01T04:17:25.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3689000000000001e-05 2023-11-01T04:17:25.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.962703+0000", 2023-11-01T04:17:25.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.099217692999999996 2023-11-01T04:17:25.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.962756+0000", 2023-11-01T04:17:25.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2454999999999998e-05 2023-11-01T04:17:25.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063680+0000", 2023-11-01T04:17:25.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100924298 2023-11-01T04:17:25.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.063726+0000", 2023-11-01T04:17:25.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5760000000000002e-05 2023-11-01T04:17:25.620 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.620 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.620 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.621 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:121 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'83, mlcod=28'78)", 2023-11-01T04:17:25.621 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:05.864836+0000", 2023-11-01T04:17:25.621 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 79.576741186000007, 2023-11-01T04:17:25.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20010170799999999, 2023-11-01T04:17:25.621 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.622 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.622 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864836+0000", 2023-11-01T04:17:25.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864836+0000", 2023-11-01T04:17:25.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864838+0000", 2023-11-01T04:17:25.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7999999999999999e-06 2023-11-01T04:17:25.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864956+0000", 2023-11-01T04:17:25.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000118293 2023-11-01T04:17:25.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864985+0000", 2023-11-01T04:17:25.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9187999999999999e-05 2023-11-01T04:17:25.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.627 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.627 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:05.864999+0000", 2023-11-01T04:17:25.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3718e-05 2023-11-01T04:17:25.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.628 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.062876+0000", 2023-11-01T04:17:25.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.197877154 2023-11-01T04:17:25.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.062909+0000", 2023-11-01T04:17:25.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3059999999999999e-05 2023-11-01T04:17:25.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.064827+0000", 2023-11-01T04:17:25.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019179209999999999 2023-11-01T04:17:25.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.064938+0000", 2023-11-01T04:17:25.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110574 2023-11-01T04:17:25.632 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.632 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.632 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.633 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:133 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'91, mlcod=28'85)", 2023-11-01T04:17:25.633 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:06.724959+0000", 2023-11-01T04:17:25.633 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 78.716617686999996, 2023-11-01T04:17:25.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.108020745, 2023-11-01T04:17:25.634 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.634 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.634 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.724959+0000", 2023-11-01T04:17:25.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.724959+0000", 2023-11-01T04:17:25.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.724960+0000", 2023-11-01T04:17:25.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3000000000000001e-07 2023-11-01T04:17:25.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.724965+0000", 2023-11-01T04:17:25.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.463e-06 2023-11-01T04:17:25.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.724978+0000", 2023-11-01T04:17:25.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2748999999999999e-05 2023-11-01T04:17:25.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.825118+0000", 2023-11-01T04:17:25.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100140121 2023-11-01T04:17:25.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.827391+0000", 2023-11-01T04:17:25.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022731800000000001 2023-11-01T04:17:25.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.827428+0000", 2023-11-01T04:17:25.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.65e-05 2023-11-01T04:17:25.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.832911+0000", 2023-11-01T04:17:25.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0054829340000000001 2023-11-01T04:17:25.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.832980+0000", 2023-11-01T04:17:25.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9368000000000004e-05 2023-11-01T04:17:25.644 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.644 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.644 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.645 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.4241.0:138 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'96, mlcod=28'85)", 2023-11-01T04:17:25.645 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:06.925635+0000", 2023-11-01T04:17:25.645 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 78.515941615000003, 2023-11-01T04:17:25.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20470132099999999, 2023-11-01T04:17:25.645 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.646 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2023-11-01T04:17:25.646 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.925635+0000", 2023-11-01T04:17:25.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.925635+0000", 2023-11-01T04:17:25.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.925636+0000", 2023-11-01T04:17:25.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1999999999999999e-07 2023-11-01T04:17:25.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.925647+0000", 2023-11-01T04:17:25.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1348999999999999e-05 2023-11-01T04:17:25.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:06.925660+0000", 2023-11-01T04:17:25.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2782000000000001e-05 2023-11-01T04:17:25.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.025800+0000", 2023-11-01T04:17:25.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.100139379 2023-11-01T04:17:25.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.025960+0000", 2023-11-01T04:17:25.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000160063 2023-11-01T04:17:25.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2023-11-01T04:17:25.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.026036+0000", 2023-11-01T04:17:25.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6637000000000005e-05 2023-11-01T04:17:25.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2023-11-01T04:17:25.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.130231+0000", 2023-11-01T04:17:25.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.104194433 2023-11-01T04:17:25.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:07.130337+0000", 2023-11-01T04:17:25.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010605800000000001 2023-11-01T04:17:25.655 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.655 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.655 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.656 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.4s1 54/31 ECSubWrite(tid=62, reqid=osd.10.0:62, at_version=54'25, trim_to=0'0, roll_forward_to=48'23))", 2023-11-01T04:17:25.656 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:16:34.226450+0000", 2023-11-01T04:17:25.656 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 51.215127432000003, 2023-11-01T04:17:25.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.20242526499999999, 2023-11-01T04:17:25.656 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.657 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:17:25.657 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.226450+0000", 2023-11-01T04:17:25.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.226450+0000", 2023-11-01T04:17:25.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.226451+0000", 2023-11-01T04:17:25.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6700000000000002e-07 2023-11-01T04:17:25.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.226501+0000", 2023-11-01T04:17:25.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0608000000000001e-05 2023-11-01T04:17:25.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.226539+0000", 2023-11-01T04:17:25.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7914999999999999e-05 2023-11-01T04:17:25.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.326654+0000", 2023-11-01T04:17:25.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10011447699999999 2023-11-01T04:17:25.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.426948+0000", 2023-11-01T04:17:25.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10029434299999999 2023-11-01T04:17:25.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:25.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.426967+0000", 2023-11-01T04:17:25.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9516000000000001e-05 2023-11-01T04:17:25.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:25.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.428805+0000", 2023-11-01T04:17:25.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001837991 2023-11-01T04:17:25.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:16:34.428875+0000", 2023-11-01T04:17:25.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9448000000000003e-05 2023-11-01T04:17:25.666 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.666 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.666 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.667 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(3.0s5 82/78 ECSubWrite(tid=81, reqid=client.4559.0:730, at_version=82'36, trim_to=0'0, roll_forward_to=0'0))", 2023-11-01T04:17:25.667 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2023-11-01T04:17:03.637613+0000", 2023-11-01T04:17:25.667 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.803963677999999, 2023-11-01T04:17:25.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10737751399999999, 2023-11-01T04:17:25.667 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2023-11-01T04:17:25.668 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2023-11-01T04:17:25.668 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2023-11-01T04:17:25.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2023-11-01T04:17:25.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.637613+0000", 2023-11-01T04:17:25.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2023-11-01T04:17:25.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.637613+0000", 2023-11-01T04:17:25.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2023-11-01T04:17:25.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2023-11-01T04:17:25.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.637614+0000", 2023-11-01T04:17:25.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7299999999999996e-07 2023-11-01T04:17:25.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2023-11-01T04:17:25.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.637633+0000", 2023-11-01T04:17:25.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9264999999999999e-05 2023-11-01T04:17:25.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2023-11-01T04:17:25.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.637648+0000", 2023-11-01T04:17:25.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4473e-05 2023-11-01T04:17:25.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2023-11-01T04:17:25.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.637657+0000", 2023-11-01T04:17:25.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1560000000000001e-06 2023-11-01T04:17:25.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2023-11-01T04:17:25.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.743214+0000", 2023-11-01T04:17:25.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.10555750799999999 2023-11-01T04:17:25.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2023-11-01T04:17:25.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.743246+0000", 2023-11-01T04:17:25.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1924999999999997e-05 2023-11-01T04:17:25.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2023-11-01T04:17:25.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.744939+0000", 2023-11-01T04:17:25.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001693216 2023-11-01T04:17:25.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-11-01T04:17:25.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2023-11-01T04:17:25.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2023-11-01T04:17:25.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2023-11-01T04:17:03.744991+0000", 2023-11-01T04:17:25.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1397999999999998e-05 2023-11-01T04:17:25.677 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.677 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.677 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.677 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-11-01T04:17:25.677 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-11-01T04:17:25.678 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-11-01T04:17:27.146 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-11-01T04:17:28.146 INFO:teuthology.orchestra.run.smithi153.stderr:noscrub is unset 2023-11-01T04:17:28.171 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-11-01T04:17:28.531 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2023-11-01T04:17:28.642 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:28.646 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:28.647 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['config', 'set', 'filestore_inject_stall', '3'] 2023-11-01T04:17:29.149 INFO:teuthology.orchestra.run.smithi153.stderr:nodeep-scrub is unset 2023-11-01T04:17:31.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~36s 2023-11-01T04:17:31.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~29s 2023-11-01T04:17:33.648 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2023-11-01T04:17:33.765 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:33.768 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:33.769 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['config', 'set', 'filestore_inject_stall', '3'] 2023-11-01T04:17:38.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~43s 2023-11-01T04:17:38.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~35s 2023-11-01T04:17:38.770 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2023-11-01T04:17:38.884 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:38.887 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:38.887 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['config', 'set', 'filestore_inject_stall', '3'] 2023-11-01T04:17:43.889 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2023-11-01T04:17:44.006 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:44.009 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['config', 'set', 'filestore_inject_stall', '3'] 2023-11-01T04:17:45.373 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~50s 2023-11-01T04:17:45.374 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~42s 2023-11-01T04:17:49.011 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2023-11-01T04:17:49.128 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:49.132 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['config', 'set', 'filestore_inject_stall', '3'] 2023-11-01T04:17:52.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~57s 2023-11-01T04:17:52.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~49s 2023-11-01T04:17:54.134 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2023-11-01T04:17:54.252 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:54.256 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:54.256 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['config', 'set', 'filestore_inject_stall', '3'] 2023-11-01T04:17:59.207 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~64s 2023-11-01T04:17:59.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~56s 2023-11-01T04:17:59.258 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2023-11-01T04:17:59.375 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:17:59.377 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:17:59.378 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['config', 'set', 'filestore_inject_stall', '3'] 2023-11-01T04:18:04.380 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2023-11-01T04:18:04.495 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:18:04.498 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:18:04.499 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['config', 'set', 'filestore_inject_stall', '3'] 2023-11-01T04:18:06.124 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~71s 2023-11-01T04:18:06.124 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~63s 2023-11-01T04:18:09.500 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2023-11-01T04:18:09.617 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:18:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:18:09.621 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['config', 'set', 'filestore_inject_stall', '3'] 2023-11-01T04:18:13.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~78s 2023-11-01T04:18:13.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~70s 2023-11-01T04:18:14.622 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok config set filestore_inject_stall 3 2023-11-01T04:18:14.742 INFO:teuthology.orchestra.run.smithi153.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2023-11-01T04:18:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-11-01T04:18:14.746 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1417, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1252, in self.inject_pause(key, File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1006, in inject_pause self.ceph_manager.set_config(the_one, **{conf_key: duration}) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2026, in set_config self.wait_run_admin_socket( File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1983, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.12 restart 2023-11-01T04:18:14.746 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1285, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1417, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1252, in self.inject_pause(key, File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1006, in inject_pause self.ceph_manager.set_config(the_one, **{conf_key: duration}) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2026, in set_config self.wait_run_admin_socket( File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1983, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.12 restart 2023-11-01T04:18:14.746 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2023-11-01T04:18:14.747 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2023-11-01T04:18:14.747 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2023-11-01T04:18:14.747 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2023-11-01T04:18:14.748 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-01T04:18:14.748 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-01T04:18:15.029 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:18:15.029 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":106,"fsid":"0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d","created":"2023-11-01T04:15:44.902210+0000","modified":"2023-11-01T04:18:05.911371+0000","last_up_change":"2023-11-01T04:16:58.831691+0000","last_in_change":"2023-11-01T04:16:52.779010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-01T04:15:57.685107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-01T04:15:59.882949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83ce3480-defe-4c73-aa1d-47c7941a23ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6817","nonce":59702962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6819","nonce":59702962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6823","nonce":59702962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6821","nonce":59702962}]},"public_addr":"172.21.15.153:6817/59702962","cluster_addr":"172.21.15.153:6819/59702962","heartbeat_back_addr":"172.21.15.153:6823/59702962","heartbeat_front_addr":"172.21.15.153:6821/59702962","state":["exists","up"]},{"osd":1,"uuid":"01a9801f-72fb-4299-8708-d70eb59b0fb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6815","nonce":3103315277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6817","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6818","nonce":3103315277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6823","nonce":3103315277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6821","nonce":3103315277}]},"public_addr":"172.21.15.173:6815/3103315277","cluster_addr":"172.21.15.173:6818/3103315277","heartbeat_back_addr":"172.21.15.173:6823/3103315277","heartbeat_front_addr":"172.21.15.173:6821/3103315277","state":["exists","up"]},{"osd":2,"uuid":"7c16db19-9bec-4ca0-88eb-bb5fb673d160","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6817","nonce":1072240131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6820","nonce":1072240131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6825","nonce":1072240131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6821","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6823","nonce":1072240131}]},"public_addr":"172.21.15.181:6817/1072240131","cluster_addr":"172.21.15.181:6820/1072240131","heartbeat_back_addr":"172.21.15.181:6825/1072240131","heartbeat_front_addr":"172.21.15.181:6823/1072240131","state":["exists","up"]},{"osd":3,"uuid":"4c4cc8d1-6761-4d20-90e8-7c3d43d889d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6827","nonce":2304315422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6829","nonce":2304315422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6833","nonce":2304315422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6831","nonce":2304315422}]},"public_addr":"172.21.15.190:6827/2304315422","cluster_addr":"172.21.15.190:6829/2304315422","heartbeat_back_addr":"172.21.15.190:6833/2304315422","heartbeat_front_addr":"172.21.15.190:6831/2304315422","state":["exists","up"]},{"osd":4,"uuid":"0fb14630-87e6-4456-9640-7986a5079821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6825","nonce":3209586931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6827","nonce":3209586931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6831","nonce":3209586931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6829","nonce":3209586931}]},"public_addr":"172.21.15.153:6825/3209586931","cluster_addr":"172.21.15.153:6827/3209586931","heartbeat_back_addr":"172.21.15.153:6831/3209586931","heartbeat_front_addr":"172.21.15.153:6829/3209586931","state":["exists","up"]},{"osd":5,"uuid":"cc715599-0461-477b-beb2-8777d6f36b22","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6825","nonce":409879231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6827","nonce":409879231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6831","nonce":409879231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6829","nonce":409879231}]},"public_addr":"172.21.15.173:6825/409879231","cluster_addr":"172.21.15.173:6827/409879231","heartbeat_back_addr":"172.21.15.173:6831/409879231","heartbeat_front_addr":"172.21.15.173:6829/409879231","state":["exists","up"]},{"osd":6,"uuid":"ca5bea1c-f309-4258-b85d-738097037962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6819","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6822","nonce":3465541770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6827","nonce":3465541770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6831","nonce":3465541770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6829","nonce":3465541770}]},"public_addr":"172.21.15.181:6822/3465541770","cluster_addr":"172.21.15.181:6827/3465541770","heartbeat_back_addr":"172.21.15.181:6831/3465541770","heartbeat_front_addr":"172.21.15.181:6829/3465541770","state":["exists","up"]},{"osd":7,"uuid":"1e80e965-2a46-468c-bcb8-49bee8dd19c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6803","nonce":1763196448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6805","nonce":1763196448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6809","nonce":1763196448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6807","nonce":1763196448}]},"public_addr":"172.21.15.190:6803/1763196448","cluster_addr":"172.21.15.190:6805/1763196448","heartbeat_back_addr":"172.21.15.190:6809/1763196448","heartbeat_front_addr":"172.21.15.190:6807/1763196448","state":["exists"]},{"osd":8,"uuid":"87179b74-818f-47df-ad70-1392cfe227e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6801","nonce":1068520389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6803","nonce":1068520389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6807","nonce":1068520389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6805","nonce":1068520389}]},"public_addr":"172.21.15.153:6801/1068520389","cluster_addr":"172.21.15.153:6803/1068520389","heartbeat_back_addr":"172.21.15.153:6807/1068520389","heartbeat_front_addr":"172.21.15.153:6805/1068520389","state":["exists","up"]},{"osd":9,"uuid":"10d3c443-1748-4a7c-8b60-b58cfd565b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6801","nonce":210565898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6803","nonce":210565898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6807","nonce":210565898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6805","nonce":210565898}]},"public_addr":"172.21.15.173:6801/210565898","cluster_addr":"172.21.15.173:6803/210565898","heartbeat_back_addr":"172.21.15.173:6807/210565898","heartbeat_front_addr":"172.21.15.173:6805/210565898","state":["exists","up"]},{"osd":10,"uuid":"81bb6569-d02b-4638-8b76-f99756d415b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6801","nonce":2007605188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6803","nonce":2007605188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6807","nonce":2007605188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6805","nonce":2007605188}]},"public_addr":"172.21.15.181:6801/2007605188","cluster_addr":"172.21.15.181:6803/2007605188","heartbeat_back_addr":"172.21.15.181:6807/2007605188","heartbeat_front_addr":"172.21.15.181:6805/2007605188","state":["exists","up"]},{"osd":11,"uuid":"b8aefe11-845f-48a4-930c-b05cdfbb460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6812","nonce":2456507663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6816","nonce":2456507663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6824","nonce":2456507663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6820","nonce":2456507663}]},"public_addr":"172.21.15.190:6812/2456507663","cluster_addr":"172.21.15.190:6816/2456507663","heartbeat_back_addr":"172.21.15.190:6824/2456507663","heartbeat_front_addr":"172.21.15.190:6820/2456507663","state":["exists","up"]},{"osd":12,"uuid":"2165437d-26ba-4e50-8646-8cedbc10da53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":68,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6809","nonce":819245857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6811","nonce":819245857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6815","nonce":819245857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6813","nonce":819245857}]},"public_addr":"172.21.15.153:6809/819245857","cluster_addr":"172.21.15.153:6811/819245857","heartbeat_back_addr":"172.21.15.153:6815/819245857","heartbeat_front_addr":"172.21.15.153:6813/819245857","state":["exists"]},{"osd":13,"uuid":"7b9f38b4-f5ff-43f1-a373-2aaed710026d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6809","nonce":3482699195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6811","nonce":3482699195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6819","nonce":3482699195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6814","nonce":3482699195}]},"public_addr":"172.21.15.173:6809/3482699195","cluster_addr":"172.21.15.173:6811/3482699195","heartbeat_back_addr":"172.21.15.173:6819/3482699195","heartbeat_front_addr":"172.21.15.173:6814/3482699195","state":["exists","up"]},{"osd":14,"uuid":"054b27d2-8a4d-4851-8af8-78b71e6a537e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6809","nonce":2780345165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6811","nonce":2780345165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6815","nonce":2780345165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6813","nonce":2780345165}]},"public_addr":"172.21.15.181:6809/2780345165","cluster_addr":"172.21.15.181:6811/2780345165","heartbeat_back_addr":"172.21.15.181:6815/2780345165","heartbeat_front_addr":"172.21.15.181:6813/2780345165","state":["exists","up"]},{"osd":15,"uuid":"e650f936-3712-47cd-9b17-d732991a27a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6813","nonce":3871145029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6815","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6817","nonce":3871145029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6823","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6825","nonce":3871145029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6819","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6821","nonce":3871145029}]},"public_addr":"172.21.15.190:6813/3871145029","cluster_addr":"172.21.15.190:6817/3871145029","heartbeat_back_addr":"172.21.15.190:6825/3871145029","heartbeat_front_addr":"172.21.15.190:6821/3871145029","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:08.233469+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:12.717062+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:03.140612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:18:05.325786+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:20.311480+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":52947,"last_purged_snaps_scrub":"2023-11-01T04:17:20.365999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:18.902496+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.028107+0000","dead_epoch":78},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:10.612247+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:08.793386+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:18:03.309728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:18:03.183755+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.650893+0000","dead_epoch":68},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:21.087831+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:13.468022+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:06.291345+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.4","osds":[9,8]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":4,"length":1},{"begin":11,"length":1},{"begin":17,"length":2},{"begin":21,"length":1},{"begin":24,"length":1},{"begin":27,"length":2},{"begin":30,"length":1},{"begin":34,"length":1},{"begin":38,"length":2},{"begin":47,"length":2},{"begin":51,"length":1},{"begin":56,"length":1},{"begin":64,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-01T04:18:18.047 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-01T04:18:18.336 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:18:18.337 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":106,"fsid":"0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d","created":"2023-11-01T04:15:44.902210+0000","modified":"2023-11-01T04:18:05.911371+0000","last_up_change":"2023-11-01T04:16:58.831691+0000","last_in_change":"2023-11-01T04:16:52.779010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-01T04:15:57.685107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-01T04:15:59.882949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83ce3480-defe-4c73-aa1d-47c7941a23ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6817","nonce":59702962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6819","nonce":59702962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6823","nonce":59702962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6821","nonce":59702962}]},"public_addr":"172.21.15.153:6817/59702962","cluster_addr":"172.21.15.153:6819/59702962","heartbeat_back_addr":"172.21.15.153:6823/59702962","heartbeat_front_addr":"172.21.15.153:6821/59702962","state":["exists","up"]},{"osd":1,"uuid":"01a9801f-72fb-4299-8708-d70eb59b0fb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6815","nonce":3103315277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6817","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6818","nonce":3103315277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6823","nonce":3103315277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6821","nonce":3103315277}]},"public_addr":"172.21.15.173:6815/3103315277","cluster_addr":"172.21.15.173:6818/3103315277","heartbeat_back_addr":"172.21.15.173:6823/3103315277","heartbeat_front_addr":"172.21.15.173:6821/3103315277","state":["exists","up"]},{"osd":2,"uuid":"7c16db19-9bec-4ca0-88eb-bb5fb673d160","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6817","nonce":1072240131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6820","nonce":1072240131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6825","nonce":1072240131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6821","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6823","nonce":1072240131}]},"public_addr":"172.21.15.181:6817/1072240131","cluster_addr":"172.21.15.181:6820/1072240131","heartbeat_back_addr":"172.21.15.181:6825/1072240131","heartbeat_front_addr":"172.21.15.181:6823/1072240131","state":["exists","up"]},{"osd":3,"uuid":"4c4cc8d1-6761-4d20-90e8-7c3d43d889d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6827","nonce":2304315422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6829","nonce":2304315422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6833","nonce":2304315422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6831","nonce":2304315422}]},"public_addr":"172.21.15.190:6827/2304315422","cluster_addr":"172.21.15.190:6829/2304315422","heartbeat_back_addr":"172.21.15.190:6833/2304315422","heartbeat_front_addr":"172.21.15.190:6831/2304315422","state":["exists","up"]},{"osd":4,"uuid":"0fb14630-87e6-4456-9640-7986a5079821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6825","nonce":3209586931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6827","nonce":3209586931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6831","nonce":3209586931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6829","nonce":3209586931}]},"public_addr":"172.21.15.153:6825/3209586931","cluster_addr":"172.21.15.153:6827/3209586931","heartbeat_back_addr":"172.21.15.153:6831/3209586931","heartbeat_front_addr":"172.21.15.153:6829/3209586931","state":["exists","up"]},{"osd":5,"uuid":"cc715599-0461-477b-beb2-8777d6f36b22","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6825","nonce":409879231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6827","nonce":409879231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6831","nonce":409879231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6829","nonce":409879231}]},"public_addr":"172.21.15.173:6825/409879231","cluster_addr":"172.21.15.173:6827/409879231","heartbeat_back_addr":"172.21.15.173:6831/409879231","heartbeat_front_addr":"172.21.15.173:6829/409879231","state":["exists","up"]},{"osd":6,"uuid":"ca5bea1c-f309-4258-b85d-738097037962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6819","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6822","nonce":3465541770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6827","nonce":3465541770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6831","nonce":3465541770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6829","nonce":3465541770}]},"public_addr":"172.21.15.181:6822/3465541770","cluster_addr":"172.21.15.181:6827/3465541770","heartbeat_back_addr":"172.21.15.181:6831/3465541770","heartbeat_front_addr":"172.21.15.181:6829/3465541770","state":["exists","up"]},{"osd":7,"uuid":"1e80e965-2a46-468c-bcb8-49bee8dd19c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6803","nonce":1763196448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6805","nonce":1763196448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6809","nonce":1763196448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6807","nonce":1763196448}]},"public_addr":"172.21.15.190:6803/1763196448","cluster_addr":"172.21.15.190:6805/1763196448","heartbeat_back_addr":"172.21.15.190:6809/1763196448","heartbeat_front_addr":"172.21.15.190:6807/1763196448","state":["exists"]},{"osd":8,"uuid":"87179b74-818f-47df-ad70-1392cfe227e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6801","nonce":1068520389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6803","nonce":1068520389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6807","nonce":1068520389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6805","nonce":1068520389}]},"public_addr":"172.21.15.153:6801/1068520389","cluster_addr":"172.21.15.153:6803/1068520389","heartbeat_back_addr":"172.21.15.153:6807/1068520389","heartbeat_front_addr":"172.21.15.153:6805/1068520389","state":["exists","up"]},{"osd":9,"uuid":"10d3c443-1748-4a7c-8b60-b58cfd565b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6801","nonce":210565898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6803","nonce":210565898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6807","nonce":210565898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6805","nonce":210565898}]},"public_addr":"172.21.15.173:6801/210565898","cluster_addr":"172.21.15.173:6803/210565898","heartbeat_back_addr":"172.21.15.173:6807/210565898","heartbeat_front_addr":"172.21.15.173:6805/210565898","state":["exists","up"]},{"osd":10,"uuid":"81bb6569-d02b-4638-8b76-f99756d415b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6801","nonce":2007605188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6803","nonce":2007605188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6807","nonce":2007605188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6805","nonce":2007605188}]},"public_addr":"172.21.15.181:6801/2007605188","cluster_addr":"172.21.15.181:6803/2007605188","heartbeat_back_addr":"172.21.15.181:6807/2007605188","heartbeat_front_addr":"172.21.15.181:6805/2007605188","state":["exists","up"]},{"osd":11,"uuid":"b8aefe11-845f-48a4-930c-b05cdfbb460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6812","nonce":2456507663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6816","nonce":2456507663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6824","nonce":2456507663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6820","nonce":2456507663}]},"public_addr":"172.21.15.190:6812/2456507663","cluster_addr":"172.21.15.190:6816/2456507663","heartbeat_back_addr":"172.21.15.190:6824/2456507663","heartbeat_front_addr":"172.21.15.190:6820/2456507663","state":["exists","up"]},{"osd":12,"uuid":"2165437d-26ba-4e50-8646-8cedbc10da53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":68,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6809","nonce":819245857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6811","nonce":819245857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6815","nonce":819245857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6813","nonce":819245857}]},"public_addr":"172.21.15.153:6809/819245857","cluster_addr":"172.21.15.153:6811/819245857","heartbeat_back_addr":"172.21.15.153:6815/819245857","heartbeat_front_addr":"172.21.15.153:6813/819245857","state":["exists"]},{"osd":13,"uuid":"7b9f38b4-f5ff-43f1-a373-2aaed710026d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6809","nonce":3482699195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6811","nonce":3482699195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6819","nonce":3482699195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6814","nonce":3482699195}]},"public_addr":"172.21.15.173:6809/3482699195","cluster_addr":"172.21.15.173:6811/3482699195","heartbeat_back_addr":"172.21.15.173:6819/3482699195","heartbeat_front_addr":"172.21.15.173:6814/3482699195","state":["exists","up"]},{"osd":14,"uuid":"054b27d2-8a4d-4851-8af8-78b71e6a537e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6809","nonce":2780345165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6811","nonce":2780345165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6815","nonce":2780345165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6813","nonce":2780345165}]},"public_addr":"172.21.15.181:6809/2780345165","cluster_addr":"172.21.15.181:6811/2780345165","heartbeat_back_addr":"172.21.15.181:6815/2780345165","heartbeat_front_addr":"172.21.15.181:6813/2780345165","state":["exists","up"]},{"osd":15,"uuid":"e650f936-3712-47cd-9b17-d732991a27a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6813","nonce":3871145029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6815","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6817","nonce":3871145029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6823","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6825","nonce":3871145029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6819","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6821","nonce":3871145029}]},"public_addr":"172.21.15.190:6813/3871145029","cluster_addr":"172.21.15.190:6817/3871145029","heartbeat_back_addr":"172.21.15.190:6825/3871145029","heartbeat_front_addr":"172.21.15.190:6821/3871145029","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:08.233469+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:12.717062+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:03.140612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:18:05.325786+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:20.311480+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":52947,"last_purged_snaps_scrub":"2023-11-01T04:17:20.365999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:18.902496+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.028107+0000","dead_epoch":78},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:10.612247+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:08.793386+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:18:03.309728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:18:03.183755+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.650893+0000","dead_epoch":68},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:21.087831+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:13.468022+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:06.291345+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.4","osds":[9,8]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":4,"length":1},{"begin":11,"length":1},{"begin":17,"length":2},{"begin":21,"length":1},{"begin":24,"length":1},{"begin":27,"length":2},{"begin":30,"length":1},{"begin":34,"length":1},{"begin":38,"length":2},{"begin":47,"length":2},{"begin":51,"length":1},{"begin":56,"length":1},{"begin":64,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-01T04:18:19.956 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~85s 2023-11-01T04:18:19.957 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~77s 2023-11-01T04:18:19.957 INFO:tasks.daemonwatchdog.daemon_watchdog:OSDThrasher failed 2023-11-01T04:18:19.957 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-11-01T04:18:21.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-01T04:18:21.656 INFO:teuthology.orchestra.run.smithi153.stdout: 2023-11-01T04:18:21.656 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":107,"fsid":"0dae0e98-9e8d-45a5-ba19-b17eb5a53b7d","created":"2023-11-01T04:15:44.902210+0000","modified":"2023-11-01T04:18:18.805460+0000","last_up_change":"2023-11-01T04:16:58.831691+0000","last_in_change":"2023-11-01T04:16:52.779010+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-01T04:15:57.685107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-01T04:15:59.882949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83ce3480-defe-4c73-aa1d-47c7941a23ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6817","nonce":59702962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6819","nonce":59702962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6823","nonce":59702962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":59702962},{"type":"v1","addr":"172.21.15.153:6821","nonce":59702962}]},"public_addr":"172.21.15.153:6817/59702962","cluster_addr":"172.21.15.153:6819/59702962","heartbeat_back_addr":"172.21.15.153:6823/59702962","heartbeat_front_addr":"172.21.15.153:6821/59702962","state":["exists","up"]},{"osd":1,"uuid":"01a9801f-72fb-4299-8708-d70eb59b0fb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6815","nonce":3103315277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6817","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6818","nonce":3103315277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6823","nonce":3103315277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":3103315277},{"type":"v1","addr":"172.21.15.173:6821","nonce":3103315277}]},"public_addr":"172.21.15.173:6815/3103315277","cluster_addr":"172.21.15.173:6818/3103315277","heartbeat_back_addr":"172.21.15.173:6823/3103315277","heartbeat_front_addr":"172.21.15.173:6821/3103315277","state":["exists","up"]},{"osd":2,"uuid":"7c16db19-9bec-4ca0-88eb-bb5fb673d160","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6817","nonce":1072240131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6820","nonce":1072240131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6825","nonce":1072240131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6821","nonce":1072240131},{"type":"v1","addr":"172.21.15.181:6823","nonce":1072240131}]},"public_addr":"172.21.15.181:6817/1072240131","cluster_addr":"172.21.15.181:6820/1072240131","heartbeat_back_addr":"172.21.15.181:6825/1072240131","heartbeat_front_addr":"172.21.15.181:6823/1072240131","state":["exists","up"]},{"osd":3,"uuid":"4c4cc8d1-6761-4d20-90e8-7c3d43d889d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6827","nonce":2304315422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6829","nonce":2304315422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6833","nonce":2304315422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2304315422},{"type":"v1","addr":"172.21.15.190:6831","nonce":2304315422}]},"public_addr":"172.21.15.190:6827/2304315422","cluster_addr":"172.21.15.190:6829/2304315422","heartbeat_back_addr":"172.21.15.190:6833/2304315422","heartbeat_front_addr":"172.21.15.190:6831/2304315422","state":["exists","up"]},{"osd":4,"uuid":"0fb14630-87e6-4456-9640-7986a5079821","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6825","nonce":3209586931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6827","nonce":3209586931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6831","nonce":3209586931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":3209586931},{"type":"v1","addr":"172.21.15.153:6829","nonce":3209586931}]},"public_addr":"172.21.15.153:6825/3209586931","cluster_addr":"172.21.15.153:6827/3209586931","heartbeat_back_addr":"172.21.15.153:6831/3209586931","heartbeat_front_addr":"172.21.15.153:6829/3209586931","state":["exists","up"]},{"osd":5,"uuid":"cc715599-0461-477b-beb2-8777d6f36b22","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6825","nonce":409879231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6827","nonce":409879231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6831","nonce":409879231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":409879231},{"type":"v1","addr":"172.21.15.173:6829","nonce":409879231}]},"public_addr":"172.21.15.173:6825/409879231","cluster_addr":"172.21.15.173:6827/409879231","heartbeat_back_addr":"172.21.15.173:6831/409879231","heartbeat_front_addr":"172.21.15.173:6829/409879231","state":["exists","up"]},{"osd":6,"uuid":"ca5bea1c-f309-4258-b85d-738097037962","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6819","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6822","nonce":3465541770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6827","nonce":3465541770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6831","nonce":3465541770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3465541770},{"type":"v1","addr":"172.21.15.181:6829","nonce":3465541770}]},"public_addr":"172.21.15.181:6822/3465541770","cluster_addr":"172.21.15.181:6827/3465541770","heartbeat_back_addr":"172.21.15.181:6831/3465541770","heartbeat_front_addr":"172.21.15.181:6829/3465541770","state":["exists","up"]},{"osd":7,"uuid":"1e80e965-2a46-468c-bcb8-49bee8dd19c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6803","nonce":1763196448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6805","nonce":1763196448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6809","nonce":1763196448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1763196448},{"type":"v1","addr":"172.21.15.190:6807","nonce":1763196448}]},"public_addr":"172.21.15.190:6803/1763196448","cluster_addr":"172.21.15.190:6805/1763196448","heartbeat_back_addr":"172.21.15.190:6809/1763196448","heartbeat_front_addr":"172.21.15.190:6807/1763196448","state":["exists"]},{"osd":8,"uuid":"87179b74-818f-47df-ad70-1392cfe227e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6801","nonce":1068520389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6803","nonce":1068520389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6807","nonce":1068520389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1068520389},{"type":"v1","addr":"172.21.15.153:6805","nonce":1068520389}]},"public_addr":"172.21.15.153:6801/1068520389","cluster_addr":"172.21.15.153:6803/1068520389","heartbeat_back_addr":"172.21.15.153:6807/1068520389","heartbeat_front_addr":"172.21.15.153:6805/1068520389","state":["exists","up"]},{"osd":9,"uuid":"10d3c443-1748-4a7c-8b60-b58cfd565b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6801","nonce":210565898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6803","nonce":210565898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6807","nonce":210565898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":210565898},{"type":"v1","addr":"172.21.15.173:6805","nonce":210565898}]},"public_addr":"172.21.15.173:6801/210565898","cluster_addr":"172.21.15.173:6803/210565898","heartbeat_back_addr":"172.21.15.173:6807/210565898","heartbeat_front_addr":"172.21.15.173:6805/210565898","state":["exists","up"]},{"osd":10,"uuid":"81bb6569-d02b-4638-8b76-f99756d415b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6801","nonce":2007605188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6803","nonce":2007605188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6807","nonce":2007605188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2007605188},{"type":"v1","addr":"172.21.15.181:6805","nonce":2007605188}]},"public_addr":"172.21.15.181:6801/2007605188","cluster_addr":"172.21.15.181:6803/2007605188","heartbeat_back_addr":"172.21.15.181:6807/2007605188","heartbeat_front_addr":"172.21.15.181:6805/2007605188","state":["exists","up"]},{"osd":11,"uuid":"b8aefe11-845f-48a4-930c-b05cdfbb460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6812","nonce":2456507663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6816","nonce":2456507663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6824","nonce":2456507663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2456507663},{"type":"v1","addr":"172.21.15.190:6820","nonce":2456507663}]},"public_addr":"172.21.15.190:6812/2456507663","cluster_addr":"172.21.15.190:6816/2456507663","heartbeat_back_addr":"172.21.15.190:6824/2456507663","heartbeat_front_addr":"172.21.15.190:6820/2456507663","state":["exists","up"]},{"osd":12,"uuid":"2165437d-26ba-4e50-8646-8cedbc10da53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":31,"down_at":68,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6809","nonce":819245857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6811","nonce":819245857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6815","nonce":819245857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":819245857},{"type":"v1","addr":"172.21.15.153:6813","nonce":819245857}]},"public_addr":"172.21.15.153:6809/819245857","cluster_addr":"172.21.15.153:6811/819245857","heartbeat_back_addr":"172.21.15.153:6815/819245857","heartbeat_front_addr":"172.21.15.153:6813/819245857","state":["exists"]},{"osd":13,"uuid":"7b9f38b4-f5ff-43f1-a373-2aaed710026d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6809","nonce":3482699195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6811","nonce":3482699195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6819","nonce":3482699195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":3482699195},{"type":"v1","addr":"172.21.15.173:6814","nonce":3482699195}]},"public_addr":"172.21.15.173:6809/3482699195","cluster_addr":"172.21.15.173:6811/3482699195","heartbeat_back_addr":"172.21.15.173:6819/3482699195","heartbeat_front_addr":"172.21.15.173:6814/3482699195","state":["exists","up"]},{"osd":14,"uuid":"054b27d2-8a4d-4851-8af8-78b71e6a537e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6809","nonce":2780345165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6811","nonce":2780345165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6815","nonce":2780345165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2780345165},{"type":"v1","addr":"172.21.15.181:6813","nonce":2780345165}]},"public_addr":"172.21.15.181:6809/2780345165","cluster_addr":"172.21.15.181:6811/2780345165","heartbeat_back_addr":"172.21.15.181:6815/2780345165","heartbeat_front_addr":"172.21.15.181:6813/2780345165","state":["exists","up"]},{"osd":15,"uuid":"e650f936-3712-47cd-9b17-d732991a27a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6813","nonce":3871145029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6815","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6817","nonce":3871145029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6823","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6825","nonce":3871145029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6819","nonce":3871145029},{"type":"v1","addr":"172.21.15.190:6821","nonce":3871145029}]},"public_addr":"172.21.15.190:6813/3871145029","cluster_addr":"172.21.15.190:6817/3871145029","heartbeat_back_addr":"172.21.15.190:6825/3871145029","heartbeat_front_addr":"172.21.15.190:6821/3871145029","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:08.233469+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:18:17.809961+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:03.140612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:18:05.325786+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:20.311480+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":52947,"last_purged_snaps_scrub":"2023-11-01T04:17:20.365999+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:18.902496+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:55.028107+0000","dead_epoch":78},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:10.612247+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:08.793386+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:18:03.309728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:18:03.183755+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:15:54.650893+0000","dead_epoch":68},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:21.087831+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:18:18.344693+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-01T04:17:06.291345+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.4","osds":[9,8]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":4,"length":1},{"begin":11,"length":1},{"begin":17,"length":2},{"begin":21,"length":1},{"begin":24,"length":1},{"begin":27,"length":2},{"begin":30,"length":1},{"begin":34,"length":1},{"begin":38,"length":2},{"begin":47,"length":2},{"begin":51,"length":1},{"begin":56,"length":1},{"begin":64,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-01T04:18:21.871 INFO:tasks.ceph.osd.0:Sent signal 15 2023-11-01T04:18:21.871 INFO:tasks.ceph.osd.4:Sent signal 15 2023-11-01T04:18:21.872 INFO:tasks.ceph.osd.8:Sent signal 15 2023-11-01T04:18:21.872 INFO:tasks.ceph.osd.1:Sent signal 15 2023-11-01T04:18:21.873 INFO:tasks.ceph.osd.5:Sent signal 15 2023-11-01T04:18:21.873 INFO:tasks.ceph.osd.9:Sent signal 15 2023-11-01T04:18:21.874 INFO:tasks.ceph.osd.13:Sent signal 15 2023-11-01T04:18:21.874 INFO:tasks.ceph.osd.2:Sent signal 15 2023-11-01T04:18:21.874 INFO:tasks.ceph.osd.6:Sent signal 15 2023-11-01T04:18:21.875 INFO:tasks.ceph.osd.10:Sent signal 15 2023-11-01T04:18:21.875 INFO:tasks.ceph.osd.14:Sent signal 15 2023-11-01T04:18:21.876 INFO:tasks.ceph.osd.3:Sent signal 15 2023-11-01T04:18:21.876 INFO:tasks.ceph.osd.11:Sent signal 15 2023-11-01T04:18:21.877 INFO:tasks.ceph.osd.15:Sent signal 15 2023-11-01T04:18:21.877 INFO:tasks.ceph.mon.a:Sent signal 15 2023-11-01T04:18:21.877 INFO:tasks.ceph.mon.b:Sent signal 15 2023-11-01T04:18:21.878 INFO:tasks.ceph.mon.c:Sent signal 15 2023-11-01T04:18:21.878 INFO:tasks.ceph.mgr.y:Sent signal 15 2023-11-01T04:18:21.879 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-11-01T04:18:21.880 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:18:21.875+0000 7f5c97276640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 35948) UID: 0 2023-11-01T04:18:21.881 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:18:21.875+0000 7f5c97276640 -1 osd.3 107 *** Got signal Terminated *** 2023-11-01T04:18:21.881 INFO:tasks.ceph.osd.3.smithi190.stderr:2023-11-01T04:18:21.875+0000 7f5c97276640 -1 osd.3 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-01T04:18:21.881 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:18:21.869+0000 7f83cdfc1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 35654) UID: 0 2023-11-01T04:18:21.881 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:18:21.869+0000 7f83cdfc1640 -1 osd.2 107 *** Got signal Terminated *** 2023-11-01T04:18:21.882 INFO:tasks.ceph.osd.2.smithi181.stderr:2023-11-01T04:18:21.869+0000 7f83cdfc1640 -1 osd.2 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-01T04:18:21.882 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:18:21.871+0000 7f07e1573640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 35677) UID: 0 2023-11-01T04:18:21.882 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:18:21.871+0000 7f07e1573640 -1 osd.1 107 *** Got signal Terminated *** 2023-11-01T04:18:21.883 INFO:tasks.ceph.osd.1.smithi173.stderr:2023-11-01T04:18:21.871+0000 7f07e1573640 -1 osd.1 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-01T04:18:21.883 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:18:21.870+0000 7fa483489640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 35770) UID: 0 2023-11-01T04:18:21.883 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:18:21.870+0000 7fa483489640 -1 osd.0 107 *** Got signal Terminated *** 2023-11-01T04:18:21.883 INFO:tasks.ceph.osd.0.smithi153.stderr:2023-11-01T04:18:21.870+0000 7fa483489640 -1 osd.0 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-01T04:18:21.884 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:18:21.873+0000 7f63a4983640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 35659) UID: 0 2023-11-01T04:18:21.884 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:18:21.873+0000 7f63a4983640 -1 osd.10 107 *** Got signal Terminated *** 2023-11-01T04:18:21.884 INFO:tasks.ceph.osd.10.smithi181.stderr:2023-11-01T04:18:21.873+0000 7f63a4983640 -1 osd.10 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-01T04:18:21.885 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:18:21.871+0000 7efe065b1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 35680) UID: 0 2023-11-01T04:18:21.885 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:18:21.871+0000 7efe065b1640 -1 osd.5 107 *** Got signal Terminated *** 2023-11-01T04:18:21.885 INFO:tasks.ceph.osd.5.smithi173.stderr:2023-11-01T04:18:21.871+0000 7efe065b1640 -1 osd.5 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-01T04:18:21.886 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:18:21.870+0000 7fd30c83e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 35773) UID: 0 2023-11-01T04:18:21.886 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:18:21.870+0000 7fd30c83e640 -1 osd.4 107 *** Got signal Terminated *** 2023-11-01T04:18:21.886 INFO:tasks.ceph.osd.4.smithi153.stderr:2023-11-01T04:18:21.870+0000 7fd30c83e640 -1 osd.4 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-01T04:18:21.887 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:18:21.875+0000 7f04e4e7a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 35953) UID: 0 2023-11-01T04:18:21.887 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:18:21.875+0000 7f04e4e7a640 -1 osd.11 107 *** Got signal Terminated *** 2023-11-01T04:18:21.887 INFO:tasks.ceph.osd.11.smithi190.stderr:2023-11-01T04:18:21.875+0000 7f04e4e7a640 -1 osd.11 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-01T04:18:21.887 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:18:21.873+0000 7f4052983640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 35657) UID: 0 2023-11-01T04:18:21.888 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:18:21.873+0000 7f4052983640 -1 osd.6 107 *** Got signal Terminated *** 2023-11-01T04:18:21.888 INFO:tasks.ceph.osd.6.smithi181.stderr:2023-11-01T04:18:21.873+0000 7f4052983640 -1 osd.6 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-01T04:18:21.888 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:18:21.871+0000 7fb28fa0f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 35682) UID: 0 2023-11-01T04:18:21.889 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:18:21.871+0000 7fb28fa0f640 -1 osd.9 107 *** Got signal Terminated *** 2023-11-01T04:18:21.889 INFO:tasks.ceph.osd.9.smithi173.stderr:2023-11-01T04:18:21.871+0000 7fb28fa0f640 -1 osd.9 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-01T04:18:21.889 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:18:21.875+0000 7f9bdf92c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 35955) UID: 0 2023-11-01T04:18:21.890 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:18:21.875+0000 7f9bdf92c640 -1 osd.15 107 *** Got signal Terminated *** 2023-11-01T04:18:21.890 INFO:tasks.ceph.osd.15.smithi190.stderr:2023-11-01T04:18:21.875+0000 7f9bdf92c640 -1 osd.15 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-01T04:18:21.891 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:18:21.871+0000 7f0bfc85b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 35684) UID: 0 2023-11-01T04:18:21.891 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:18:21.871+0000 7f0bfc85b640 -1 osd.13 107 *** Got signal Terminated *** 2023-11-01T04:18:21.891 INFO:tasks.ceph.osd.13.smithi173.stderr:2023-11-01T04:18:21.871+0000 7f0bfc85b640 -1 osd.13 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-01T04:18:21.891 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:18:21.870+0000 7f767a1e2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 35775) UID: 0 2023-11-01T04:18:21.892 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:18:21.870+0000 7f767a1e2640 -1 osd.8 107 *** Got signal Terminated *** 2023-11-01T04:18:21.892 INFO:tasks.ceph.osd.8.smithi153.stderr:2023-11-01T04:18:21.870+0000 7f767a1e2640 -1 osd.8 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-01T04:18:21.892 INFO:tasks.ceph.mgr.x.smithi190.stderr:2023-11-01T04:18:21.879+0000 7f06f30af640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 35516) UID: 0 2023-11-01T04:18:21.893 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:18:21.873+0000 7fc4f431f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 35661) UID: 0 2023-11-01T04:18:21.893 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:18:21.873+0000 7fc4f431f640 -1 osd.14 107 *** Got signal Terminated *** 2023-11-01T04:18:21.893 INFO:tasks.ceph.osd.14.smithi181.stderr:2023-11-01T04:18:21.873+0000 7fc4f431f640 -1 osd.14 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-01T04:18:21.893 INFO:tasks.ceph.mon.b.smithi173.stderr:2023-11-01T04:18:21.875+0000 7fa859d57640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 35625) UID: 0 2023-11-01T04:18:21.894 INFO:tasks.ceph.mon.b.smithi173.stderr:2023-11-01T04:18:21.875+0000 7fa859d57640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-11-01T04:18:21.894 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-11-01T04:18:21.874+0000 7f8fa4ead640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 35658) UID: 0 2023-11-01T04:18:21.894 INFO:tasks.ceph.mon.a.smithi153.stderr:2023-11-01T04:18:21.874+0000 7f8fa4ead640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-11-01T04:18:21.894 INFO:tasks.ceph.mon.c.smithi181.stderr:2023-11-01T04:18:21.873+0000 7fa9f96bf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 35602) UID: 0 2023-11-01T04:18:21.895 INFO:tasks.ceph.mon.c.smithi181.stderr:2023-11-01T04:18:21.873+0000 7fa9f96bf640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-11-01T04:18:22.080 INFO:tasks.ceph.mgr.y.smithi153.stderr:daemon-helper: command crashed with signal 15 2023-11-01T04:18:24.673 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-01T04:20:24.695 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-01T04:20:24.696 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2023-11-01T04:20:24.696 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-01T04:20:24.706 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-01T04:20:24.706 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-01T04:22:24.725 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-01T04:22:24.727 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-01T04:22:24.728 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-01T04:22:24.728 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-01T04:22:24.729 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-01T04:22:24.729 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.730 INFO:tasks.ceph.osd.0:Stopped 2023-11-01T04:22:24.730 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-11-01T04:22:24.731 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.731 INFO:tasks.ceph.osd.4:Stopped 2023-11-01T04:22:24.732 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2023-11-01T04:22:24.732 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.733 INFO:tasks.ceph.osd.8:Stopped 2023-11-01T04:22:24.734 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2023-11-01T04:22:24.734 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:22:24.735 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12' 2023-11-01T04:22:24.736 INFO:tasks.ceph.osd.12:Stopped 2023-11-01T04:22:24.736 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-01T04:22:24.737 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.737 INFO:tasks.ceph.osd.1:Stopped 2023-11-01T04:22:24.738 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-11-01T04:22:24.738 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.739 INFO:tasks.ceph.osd.5:Stopped 2023-11-01T04:22:24.740 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2023-11-01T04:22:24.740 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.741 INFO:tasks.ceph.osd.9:Stopped 2023-11-01T04:22:24.741 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2023-11-01T04:22:24.741 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.742 INFO:tasks.ceph.osd.13:Stopped 2023-11-01T04:22:24.743 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-01T04:22:24.743 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.744 INFO:tasks.ceph.osd.2:Stopped 2023-11-01T04:22:24.744 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-11-01T04:22:24.745 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.745 INFO:tasks.ceph.osd.6:Stopped 2023-11-01T04:22:24.746 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2023-11-01T04:22:24.746 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.747 INFO:tasks.ceph.osd.10:Stopped 2023-11-01T04:22:24.747 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2023-11-01T04:22:24.748 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.748 INFO:tasks.ceph.osd.14:Stopped 2023-11-01T04:22:24.749 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-11-01T04:22:24.749 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.750 INFO:tasks.ceph.osd.3:Stopped 2023-11-01T04:22:24.750 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-11-01T04:22:24.751 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:22:24.752 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7' 2023-11-01T04:22:24.753 INFO:tasks.ceph.osd.7:Stopped 2023-11-01T04:22:24.753 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2023-11-01T04:22:24.754 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.754 INFO:tasks.ceph.osd.11:Stopped 2023-11-01T04:22:24.755 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2023-11-01T04:22:24.755 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.756 INFO:tasks.ceph.osd.15:Stopped 2023-11-01T04:22:24.757 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-01T04:22:24.757 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-11-01T04:22:24.758 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:22:24.759 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y' 2023-11-01T04:22:24.759 INFO:tasks.ceph.mgr.y:Stopped 2023-11-01T04:22:24.760 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-01T04:22:24.760 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.761 INFO:tasks.ceph.mgr.x:Stopped 2023-11-01T04:22:24.762 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-01T04:22:24.762 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-01T04:22:24.763 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.763 INFO:tasks.ceph.mon.a:Stopped 2023-11-01T04:22:24.764 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-01T04:22:24.764 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.765 INFO:tasks.ceph.mon.b:Stopped 2023-11-01T04:22:24.765 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-01T04:22:24.766 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T04:22:24.766 INFO:tasks.ceph.mon.c:Stopped 2023-11-01T04:22:24.767 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-01T04:22:24.768 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | head -n 1 2023-11-01T04:22:24.786 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi153.front.sepia.ceph.com 2023-11-01T04:22:24.787 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-01T04:22:24.962 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi153.front.sepia.ceph.com 2023-11-01T04:22:24.963 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-11-01T04:22:25.099 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi153.front.sepia.ceph.com 2023-11-01T04:22:25.099 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2023-11-01T04:22:25.283 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi153.front.sepia.ceph.com 2023-11-01T04:22:25.284 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2023-11-01T04:22:25.474 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi173.front.sepia.ceph.com 2023-11-01T04:22:25.475 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-01T04:22:25.657 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi173.front.sepia.ceph.com 2023-11-01T04:22:25.658 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-11-01T04:22:25.797 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi173.front.sepia.ceph.com 2023-11-01T04:22:25.798 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2023-11-01T04:22:25.939 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi173.front.sepia.ceph.com 2023-11-01T04:22:25.939 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2023-11-01T04:22:26.066 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi181.front.sepia.ceph.com 2023-11-01T04:22:26.067 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-01T04:22:26.238 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi181.front.sepia.ceph.com 2023-11-01T04:22:26.239 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-11-01T04:22:26.380 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi181.front.sepia.ceph.com 2023-11-01T04:22:26.381 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2023-11-01T04:22:26.556 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi181.front.sepia.ceph.com 2023-11-01T04:22:26.557 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2023-11-01T04:22:26.714 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi190.front.sepia.ceph.com 2023-11-01T04:22:26.715 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-11-01T04:22:27.033 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi190.front.sepia.ceph.com 2023-11-01T04:22:27.034 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-11-01T04:22:27.209 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi190.front.sepia.ceph.com 2023-11-01T04:22:27.210 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2023-11-01T04:22:27.386 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi190.front.sepia.ceph.com 2023-11-01T04:22:27.386 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2023-11-01T04:22:27.550 INFO:tasks.ceph:Archiving mon data... 2023-11-01T04:22:27.552 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312/data/mon.a.tgz 2023-11-01T04:22:27.553 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2023-11-01T04:22:27.559 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.eUe8xVpcHK 2023-11-01T04:22:27.560 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.eUe8xVpcHK 2023-11-01T04:22:27.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.eUe8xVpcHK 2023-11-01T04:22:27.759 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.eUe8xVpcHK is 367KB 2023-11-01T04:22:27.821 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.eUe8xVpcHK 2023-11-01T04:22:27.826 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312/data/mon.b.tgz 2023-11-01T04:22:27.827 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2023-11-01T04:22:27.833 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.K9j9TTOlob 2023-11-01T04:22:27.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.K9j9TTOlob 2023-11-01T04:22:27.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.K9j9TTOlob 2023-11-01T04:22:28.059 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.K9j9TTOlob is 502KB 2023-11-01T04:22:28.125 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.K9j9TTOlob 2023-11-01T04:22:28.131 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312/data/mon.c.tgz 2023-11-01T04:22:28.131 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2023-11-01T04:22:28.138 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.A3Hi9dVcSZ 2023-11-01T04:22:28.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.A3Hi9dVcSZ 2023-11-01T04:22:28.303 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.A3Hi9dVcSZ 2023-11-01T04:22:28.367 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.A3Hi9dVcSZ is 502KB 2023-11-01T04:22:28.429 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.A3Hi9dVcSZ 2023-11-01T04:22:28.435 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-01T04:22:28.435 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-01T04:22:28.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-01T04:22:28.441 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-01T04:22:28.480 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-01T04:22:28.495 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-01T04:22:28.495 DEBUG:teuthology.orchestra.run.smithi153:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-01T04:22:28.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-01T04:22:28.501 DEBUG:teuthology.orchestra.run.smithi181:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-01T04:22:28.515 INFO:teuthology.orchestra.run.smithi173.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-01T04:22:28.525 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-01T04:22:28.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-01T04:22:28.573 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-01T04:22:28.584 INFO:teuthology.orchestra.run.smithi190.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-01T04:22:28.587 INFO:tasks.ceph:Archiving crash dumps... 2023-11-01T04:22:28.589 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/crash to /home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312/remote/smithi153/crash 2023-11-01T04:22:28.590 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-01T04:22:28.638 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/crash to /home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312/remote/smithi173/crash 2023-11-01T04:22:28.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-01T04:22:28.655 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/crash to /home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312/remote/smithi181/crash 2023-11-01T04:22:28.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-01T04:22:28.668 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/crash to /home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312/remote/smithi190/crash 2023-11-01T04:22:28.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-01T04:22:28.712 INFO:tasks.ceph:Compressing logs... 2023-11-01T04:22:28.713 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-01T04:22:28.716 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-01T04:22:28.718 DEBUG:teuthology.orchestra.run.smithi181:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-01T04:22:28.721 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-01T04:22:28.729 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41093.log 2023-11-01T04:22:28.730 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41391.log 2023-11-01T04:22:28.730 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42228.log 2023-11-01T04:22:28.731 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41093.log.gz 2023-11-01T04:22:28.732 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5/var/log/ceph/ceph-client.admin.41391.log: --verbose -- /var/log/ceph/ceph-client.admin.39645.log 2023-11-01T04:22:28.732 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41391.log.gz 2023-11-01T04:22:28.733 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42228.log.gz 2023-11-01T04:22:28.733 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42275.log 2023-11-01T04:22:28.733 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39253.log 2023-11-01T04:22:28.733 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39645.log: /var/log/ceph/ceph-client.admin.42275.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-11-01T04:22:28.733 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39645.log.gz 2023-11-01T04:22:28.734 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42275.log.gz 2023-11-01T04:22:28.734 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39149.log 0.0% 2023-11-01T04:22:28.734 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39253.log.gz 2023-11-01T04:22:28.737 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43285.log 2023-11-01T04:22:28.737 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40195.log 2023-11-01T04:22:28.737 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39149.log.gz 2023-11-01T04:22:28.738 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43285.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-11-01T04:22:28.738 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43285.log.gz 2023-11-01T04:22:28.738 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35794.log 2023-11-01T04:22:28.738 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40195.log.gz 2023-11-01T04:22:28.738 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41005.log 2023-11-01T04:22:28.739 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43915.log 2023-11-01T04:22:28.739 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35794.log.gz 2023-11-01T04:22:28.739 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41005.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41005.log.gz 2023-11-01T04:22:28.739 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41966.log 2023-11-01T04:22:28.739 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39285.log 2023-11-01T04:22:28.740 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43915.log.gz 2023-11-01T04:22:28.740 INFO:teuthology.orchestra.run.smithi153.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-11-01T04:22:28.740 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41966.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41966.log.gz 2023-11-01T04:22:28.740 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42400.log 2023-11-01T04:22:28.740 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41274.log 2023-11-01T04:22:28.740 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39285.log.gz 2023-11-01T04:22:28.741 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39699.log 2023-11-01T04:22:28.741 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42400.log.gz 2023-11-01T04:22:28.741 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41274.log.gz 2023-11-01T04:22:28.741 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39283.log 2023-11-01T04:22:28.746 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-11-01T04:22:28.746 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-11-01T04:22:28.746 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2023-11-01T04:22:28.746 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-01T04:22:28.747 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-11-01T04:22:28.747 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.14.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-01T04:22:28.747 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2023-11-01T04:22:28.747 INFO:teuthology.orchestra.run.smithi181.stderr: 85.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-01T04:22:28.747 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.10.log: /var/log/ceph/ceph.audit.log: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-01T04:22:28.748 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-11-01T04:22:28.748 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-01T04:22:28.749 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-01T04:22:28.749 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-01T04:22:28.749 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2023-11-01T04:22:28.749 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2023-11-01T04:22:28.750 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-osd.9.log: 85.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-01T04:22:28.750 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-11-01T04:22:28.750 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph-osd.13.log: /var/log/ceph/ceph-mon.b.log: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-01T04:22:28.784 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35672.log 2023-11-01T04:22:28.784 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-11-01T04:22:28.784 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35672.log.gz 2023-11-01T04:22:28.784 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.39307.log 2023-11-01T04:22:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35799.log 2023-11-01T04:22:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.0.39307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35648.log 2023-11-01T04:22:28.785 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.39307.log.gz 2023-11-01T04:22:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35799.log.gz 2023-11-01T04:22:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35600.log 2023-11-01T04:22:28.785 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35648.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-11-01T04:22:28.786 INFO:teuthology.orchestra.run.smithi190.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35648.log.gz 2023-11-01T04:22:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35600.log.gz 2023-11-01T04:22:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2023-11-01T04:22:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2023-11-01T04:22:28.786 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-01T04:22:28.787 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35922.log 2023-11-01T04:22:28.787 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35576.log 2023-11-01T04:22:28.787 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35922.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35922.log.gz 2023-11-01T04:22:28.787 INFO:teuthology.orchestra.run.smithi190.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35775.log 2023-11-01T04:22:28.787 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35576.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35576.log.gz 2023-11-01T04:22:28.787 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35727.log 2023-11-01T04:22:28.788 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35775.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35775.log.gz 2023-11-01T04:22:28.788 INFO:teuthology.orchestra.run.smithi190.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35624.log 2023-11-01T04:22:28.788 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35727.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35727.log.gz 2023-11-01T04:22:28.788 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35898.log 2023-11-01T04:22:28.788 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35624.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35624.log.gz 2023-11-01T04:22:28.789 INFO:teuthology.orchestra.run.smithi190.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35552.log 2023-11-01T04:22:28.789 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35898.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35898.log.gz 2023-11-01T04:22:28.789 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35823.log 2023-11-01T04:22:28.789 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35552.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35552.log.gz 2023-11-01T04:22:28.789 INFO:teuthology.orchestra.run.smithi190.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35751.log 2023-11-01T04:22:28.789 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35823.log.gz 2023-11-01T04:22:28.790 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35849.log 2023-11-01T04:22:28.790 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35751.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35751.log.gz 2023-11-01T04:22:28.790 INFO:teuthology.orchestra.run.smithi190.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35874.log 2023-11-01T04:22:28.790 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35849.log.gz 2023-11-01T04:22:28.791 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35696.log 2023-11-01T04:22:28.791 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35874.log.gz 2023-11-01T04:22:28.791 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.35696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35696.log.gz 2023-11-01T04:22:28.798 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40225.log 2023-11-01T04:22:28.799 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39699.log.gz 2023-11-01T04:22:28.799 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41841.log 2023-11-01T04:22:28.799 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39283.log.gz 2023-11-01T04:22:28.799 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39687.log 2023-11-01T04:22:28.799 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40225.log.gz 2023-11-01T04:22:28.799 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41595.log 2023-11-01T04:22:28.800 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41841.log.gz 2023-11-01T04:22:28.800 INFO:teuthology.orchestra.run.smithi153.stderr:gzip/var/log/ceph/ceph-client.admin.39687.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41460.log 2023-11-01T04:22:28.800 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39687.log.gz 2023-11-01T04:22:28.800 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41595.log.gz 2023-11-01T04:22:28.800 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41092.log 2023-11-01T04:22:28.800 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44010.log 2023-11-01T04:22:28.801 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41460.log.gz 2023-11-01T04:22:28.801 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43606.log 2023-11-01T04:22:28.801 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41092.log.gz 2023-11-01T04:22:28.801 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.44010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44010.log.gz 2023-11-01T04:22:28.801 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39279.log 2023-11-01T04:22:28.801 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41348.log 2023-11-01T04:22:28.802 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43606.log.gz 2023-11-01T04:22:28.802 INFO:teuthology.orchestra.run.smithi153.stderr:gzip/var/log/ceph/ceph-client.admin.39279.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43311.log 2023-11-01T04:22:28.802 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39279.log.gz 2023-11-01T04:22:28.802 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41348.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2023-11-01T04:22:28.802 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41348.log.gz 2023-11-01T04:22:28.803 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41809.log 2023-11-01T04:22:28.803 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43311.log.gz 2023-11-01T04:22:28.803 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41621.log 2023-11-01T04:22:28.803 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39675.log 2023-11-01T04:22:28.803 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41809.log.gz 2023-11-01T04:22:28.803 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42133.log 2023-11-01T04:22:28.804 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41621.log.gz 2023-11-01T04:22:28.804 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39675.log.gz 2023-11-01T04:22:28.804 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39072.log 2023-11-01T04:22:28.804 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43665.log 2023-11-01T04:22:28.804 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42133.log.gz 2023-11-01T04:22:28.804 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39072.log.gz 2023-11-01T04:22:28.805 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43669.log 2023-11-01T04:22:28.805 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43665.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-01T04:22:28.805 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43665.log.gz 2023-11-01T04:22:28.805 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43669.log.gz 2023-11-01T04:22:28.805 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42974.log 2023-11-01T04:22:28.805 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42913.log 2023-11-01T04:22:28.806 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42974.log.gz 2023-11-01T04:22:28.806 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35907.log 2023-11-01T04:22:28.806 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43481.log 2023-11-01T04:22:28.806 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42913.log.gz 2023-11-01T04:22:28.806 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42504.log 2023-11-01T04:22:28.806 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35907.log.gz 2023-11-01T04:22:28.807 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43481.log.gz 2023-11-01T04:22:28.807 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40263.log 2023-11-01T04:22:28.807 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42646.log 2023-11-01T04:22:28.807 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42504.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42504.log.gz 2023-11-01T04:22:28.807 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40498.log 2023-11-01T04:22:28.807 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40263.log.gz 2023-11-01T04:22:28.808 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42646.log.gz 2023-11-01T04:22:28.808 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40574.log 2023-11-01T04:22:28.808 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39641.log 2023-11-01T04:22:28.808 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40498.log: /var/log/ceph/ceph-client.admin.40574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40498.log.gz 2023-11-01T04:22:28.808 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40574.log.gz 2023-11-01T04:22:28.808 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43019.log 2023-11-01T04:22:28.809 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40741.log 2023-11-01T04:22:28.809 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39641.log: /var/log/ceph/ceph-client.admin.43019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39641.log.gz 2023-11-01T04:22:28.809 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43019.log.gz 2023-11-01T04:22:28.809 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39673.log 2023-11-01T04:22:28.809 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41992.log 2023-11-01T04:22:28.809 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40741.log.gz 2023-11-01T04:22:28.810 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39673.log.gz 2023-11-01T04:22:28.810 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41063.log 2023-11-01T04:22:28.810 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43993.log 2023-11-01T04:22:28.810 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41992.log.gz 2023-11-01T04:22:28.810 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39702.log 2023-11-01T04:22:28.811 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41063.log.gz 2023-11-01T04:22:28.811 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43993.log.gz 2023-11-01T04:22:28.811 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39278.log 2023-11-01T04:22:28.811 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43518.log 2023-11-01T04:22:28.811 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39702.log: /var/log/ceph/ceph-client.admin.39278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39702.log.gz 2023-11-01T04:22:28.811 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39278.log.gz 2023-11-01T04:22:28.812 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41434.log 2023-11-01T04:22:28.812 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40267.log 2023-11-01T04:22:28.812 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43518.log.gz 2023-11-01T04:22:28.812 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40869.log 2023-11-01T04:22:28.812 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41434.log.gz 2023-11-01T04:22:28.812 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40267.log.gz 2023-11-01T04:22:28.813 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39656.log 2023-11-01T04:22:28.813 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39286.log 2023-11-01T04:22:28.813 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40869.log.gz 2023-11-01T04:22:28.813 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43581.log 2023-11-01T04:22:28.813 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39656.log.gz 2023-11-01T04:22:28.813 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39286.log.gz 2023-11-01T04:22:28.814 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40059.log 2023-11-01T04:22:28.814 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40663.log 2023-11-01T04:22:28.814 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43581.log.gz 2023-11-01T04:22:28.814 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40059.log.gz 2023-11-01T04:22:28.814 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35665.log 2023-11-01T04:22:28.814 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43761.log 2023-11-01T04:22:28.815 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40663.log: /var/log/ceph/ceph-client.admin.35665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40663.log.gz 2023-11-01T04:22:28.815 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35665.log.gz 2023-11-01T04:22:28.815 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41228.log 2023-11-01T04:22:28.815 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41814.log 2023-11-01T04:22:28.815 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43761.log: 0.0%/var/log/ceph/ceph-client.admin.41228.log: -- replaced with /var/log/ceph/ceph-client.admin.43761.log.gz 2023-11-01T04:22:28.815 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41228.log.gz 2023-11-01T04:22:28.816 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41730.log 2023-11-01T04:22:28.816 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42533.log 2023-11-01T04:22:28.816 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41814.log: /var/log/ceph/ceph-client.admin.41730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41814.log.gz 2023-11-01T04:22:28.816 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41730.log.gz 2023-11-01T04:22:28.816 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30889.log 2023-11-01T04:22:28.817 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39097.log 2023-11-01T04:22:28.817 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42533.log.gz 2023-11-01T04:22:28.817 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.tmp-client.admin.30889.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30889.log.gz 2023-11-01T04:22:28.817 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43172.log 2023-11-01T04:22:28.817 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39254.log 2023-11-01T04:22:28.817 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39097.log.gz 2023-11-01T04:22:28.818 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43172.log.gz 2023-11-01T04:22:28.818 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41139.log 2023-11-01T04:22:28.818 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40823.log 2023-11-01T04:22:28.818 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39254.log.gz 2023-11-01T04:22:28.818 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40088.log 2023-11-01T04:22:28.818 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41139.log.gz 2023-11-01T04:22:28.819 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41768.log 2023-11-01T04:22:28.819 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40823.log.gz 2023-11-01T04:22:28.819 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40088.log.gz 2023-11-01T04:22:28.819 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42827.log 2023-11-01T04:22:28.819 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41510.log 2023-11-01T04:22:28.820 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41768.log.gz 2023-11-01T04:22:28.820 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42827.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42827.log.gz 2023-11-01T04:22:28.820 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39201.log 2023-11-01T04:22:28.820 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39676.log 2023-11-01T04:22:28.820 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41510.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41510.log.gz 2023-11-01T04:22:28.820 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39201.log.gz 2023-11-01T04:22:28.821 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40441.log 2023-11-01T04:22:28.821 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43967.log 2023-11-01T04:22:28.821 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39676.log.gz 2023-11-01T04:22:28.821 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41306.log 2023-11-01T04:22:28.821 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40441.log.gz 2023-11-01T04:22:28.821 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43967.log.gz 2023-11-01T04:22:28.822 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42771.log 2023-11-01T04:22:28.822 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39690.log 2023-11-01T04:22:28.822 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41306.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41306.log.gz 2023-11-01T04:22:28.822 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42771.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-01T04:22:28.822 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42771.log.gz 2023-11-01T04:22:28.823 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42592.log 2023-11-01T04:22:28.823 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39690.log.gz 2023-11-01T04:22:28.823 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42050.log 2023-11-01T04:22:28.823 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42592.log.gz 2023-11-01T04:22:28.823 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39255.log 2023-11-01T04:22:28.823 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39123.log 2023-11-01T04:22:28.824 INFO:teuthology.orchestra.run.smithi153.stderr: 85.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-01T04:22:28.824 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42050.log.gz 2023-11-01T04:22:28.824 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43674.log 2023-11-01T04:22:28.824 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39255.log.gz 2023-11-01T04:22:28.824 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39123.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39123.log.gz 2023-11-01T04:22:28.824 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42743.log 2023-11-01T04:22:28.825 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43849.log 2023-11-01T04:22:28.825 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43674.log.gz 2023-11-01T04:22:28.825 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42743.log.gz 2023-11-01T04:22:28.825 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40169.log 2023-11-01T04:22:28.825 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40980.log 2023-11-01T04:22:28.825 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43849.log.gz 2023-11-01T04:22:28.826 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42426.log 2023-11-01T04:22:28.826 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40169.log.gz 2023-11-01T04:22:28.826 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41629.log 2023-11-01T04:22:28.826 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40980.log.gz 2023-11-01T04:22:28.826 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42426.log.gz 2023-11-01T04:22:28.826 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40269.log 2023-11-01T04:22:28.827 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39652.log 2023-11-01T04:22:28.827 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41629.log.gz 2023-11-01T04:22:28.827 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42343.log 2023-11-01T04:22:28.827 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40269.log.gz 2023-11-01T04:22:28.827 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39652.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-01T04:22:28.827 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39652.log.gz 2023-11-01T04:22:28.828 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42343.log.gz 2023-11-01T04:22:28.828 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40493.log 2023-11-01T04:22:28.828 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43941.log 2023-11-01T04:22:28.828 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40493.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40493.log.gz 2023-11-01T04:22:28.828 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39269.log 2023-11-01T04:22:28.828 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42018.log 2023-11-01T04:22:28.829 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43941.log.gz 2023-11-01T04:22:28.829 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39262.log 2023-11-01T04:22:28.829 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39269.log.gz 2023-11-01T04:22:28.829 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42018.log.gz 2023-11-01T04:22:28.829 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39251.log 2023-11-01T04:22:28.829 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-01T04:22:28.830 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.39262.log: /var/log/ceph/ceph-osd.12.log 2023-11-01T04:22:28.830 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39262.log.gz 2023-11-01T04:22:28.830 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39175.log 2023-11-01T04:22:28.830 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39251.log.gz 2023-11-01T04:22:28.830 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39678.log 2023-11-01T04:22:28.831 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43369.log 2023-11-01T04:22:28.831 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39175.log.gz 2023-11-01T04:22:28.831 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43103.log 2023-11-01T04:22:28.831 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39678.log.gz 2023-11-01T04:22:28.831 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43223.log 2023-11-01T04:22:28.831 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43369.log.gz 2023-11-01T04:22:28.832 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41464.log 2023-11-01T04:22:28.832 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43103.log.gz 2023-11-01T04:22:28.832 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40659.log 2023-11-01T04:22:28.832 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43223.log.gz 2023-11-01T04:22:28.832 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39280.log 2023-11-01T04:22:28.833 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41464.log.gz 2023-11-01T04:22:28.833 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39692.log 2023-11-01T04:22:28.833 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40659.log.gz 2023-11-01T04:22:28.833 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40394.log 2023-11-01T04:22:28.833 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39280.log.gz 2023-11-01T04:22:28.833 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42687.log 2023-11-01T04:22:28.834 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39692.log.gz 2023-11-01T04:22:28.834 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42858.log 2023-11-01T04:22:28.834 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40394.log.gz 2023-11-01T04:22:28.834 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41569.log 2023-11-01T04:22:28.834 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42687.log.gz 2023-11-01T04:22:28.834 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39276.log 2023-11-01T04:22:28.835 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42858.log.gz 2023-11-01T04:22:28.835 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40117.log 2023-11-01T04:22:28.835 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41569.log: /var/log/ceph/ceph-client.admin.39276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41569.log.gz 2023-11-01T04:22:28.835 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40143.log 2023-11-01T04:22:28.835 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39276.log.gz 2023-11-01T04:22:28.835 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40117.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40117.log.gz 2023-11-01T04:22:28.836 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42046.log 2023-11-01T04:22:28.836 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41628.log 2023-11-01T04:22:28.836 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40143.log.gz 2023-11-01T04:22:28.836 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39260.log 2023-11-01T04:22:28.836 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42046.log.gz 2023-11-01T04:22:28.836 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40930.log 2023-11-01T04:22:28.837 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41628.log.gz 2023-11-01T04:22:28.837 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39260.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-11-01T04:22:28.837 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39260.log.gz 2023-11-01T04:22:28.837 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41932.log 2023-11-01T04:22:28.837 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40930.log.gz 2023-11-01T04:22:28.837 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43394.log 2023-11-01T04:22:28.838 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42452.log 2023-11-01T04:22:28.838 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41932.log.gz 2023-11-01T04:22:28.838 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39287.log 2023-11-01T04:22:28.838 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43394.log.gz 2023-11-01T04:22:28.838 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40929.log 2023-11-01T04:22:28.839 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42452.log: /var/log/ceph/ceph-client.admin.39287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42129.log 2023-11-01T04:22:28.839 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42452.log.gz 2023-11-01T04:22:28.839 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39287.log.gz 2023-11-01T04:22:28.839 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40929.log.gz 2023-11-01T04:22:28.839 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39705.log 2023-11-01T04:22:28.840 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40580.log 2023-11-01T04:22:28.840 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.42129.log: /var/log/ceph/ceph-client.admin.39705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41166.log 2023-11-01T04:22:28.840 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42129.log.gz 2023-11-01T04:22:28.840 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39705.log.gz 2023-11-01T04:22:28.840 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40580.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40580.log.gz 2023-11-01T04:22:28.840 INFO:teuthology.orchestra.run.smithi153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43796.log 2023-11-01T04:22:28.841 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39685.log 2023-11-01T04:22:28.841 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.41166.log: /var/log/ceph/ceph-client.admin.43796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43823.log 2023-11-01T04:22:28.841 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41166.log.gz 2023-11-01T04:22:28.841 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43796.log.gz 2023-11-01T04:22:28.841 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.39685.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39685.log.gz 2023-11-01T04:22:28.841 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39695.log 2023-11-01T04:22:28.842 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43073.log 2023-11-01T04:22:28.842 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43823.log: /var/log/ceph/ceph-client.admin.39695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40709.log 2023-11-01T04:22:28.842 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43823.log.gz 2023-11-01T04:22:28.842 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39695.log.gz 2023-11-01T04:22:28.842 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.43073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43073.log.gz 2023-11-01T04:22:28.843 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39259.log 2023-11-01T04:22:28.843 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.40709.log: 0.0%/var/log/ceph/ceph-client.admin.39259.log: -- replaced with /var/log/ceph/ceph-client.admin.40709.log.gz 2023-11-01T04:22:28.843 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39259.log.gz 2023-11-01T04:22:28.843 INFO:teuthology.orchestra.run.smithi190.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-01T04:22:28.844 INFO:teuthology.orchestra.run.smithi173.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-11-01T04:22:28.867 INFO:teuthology.orchestra.run.smithi181.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-11-01T04:22:28.986 INFO:teuthology.orchestra.run.smithi153.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2023-11-01T04:22:29.009 INFO:teuthology.orchestra.run.smithi153.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-01T04:22:29.016 INFO:teuthology.orchestra.run.smithi190.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2023-11-01T04:22:29.041 INFO:teuthology.orchestra.run.smithi173.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2023-11-01T04:22:29.064 INFO:teuthology.orchestra.run.smithi173.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-11-01T04:22:29.073 INFO:teuthology.orchestra.run.smithi181.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-01T04:22:29.081 INFO:teuthology.orchestra.run.smithi153.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2023-11-01T04:22:29.083 INFO:teuthology.orchestra.run.smithi190.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-11-01T04:22:29.101 INFO:teuthology.orchestra.run.smithi181.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2023-11-01T04:22:29.126 INFO:teuthology.orchestra.run.smithi173.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-01T04:22:29.128 INFO:teuthology.orchestra.run.smithi181.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-11-01T04:22:29.131 INFO:teuthology.orchestra.run.smithi153.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-01T04:22:29.162 INFO:teuthology.orchestra.run.smithi181.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2023-11-01T04:22:29.164 INFO:teuthology.orchestra.run.smithi181.stderr: 2023-11-01T04:22:29.164 INFO:teuthology.orchestra.run.smithi181.stderr:real 0m0.440s 2023-11-01T04:22:29.164 INFO:teuthology.orchestra.run.smithi181.stderr:user 0m1.610s 2023-11-01T04:22:29.164 INFO:teuthology.orchestra.run.smithi181.stderr:sys 0m0.063s 2023-11-01T04:22:29.252 INFO:teuthology.orchestra.run.smithi190.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2023-11-01T04:22:29.271 INFO:teuthology.orchestra.run.smithi190.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-11-01T04:22:29.273 INFO:teuthology.orchestra.run.smithi190.stderr: 2023-11-01T04:22:29.273 INFO:teuthology.orchestra.run.smithi190.stderr:real 0m0.514s 2023-11-01T04:22:29.273 INFO:teuthology.orchestra.run.smithi190.stderr:user 0m1.590s 2023-11-01T04:22:29.273 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m0.051s 2023-11-01T04:22:29.288 INFO:teuthology.orchestra.run.smithi173.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2023-11-01T04:22:29.290 INFO:teuthology.orchestra.run.smithi173.stderr: 2023-11-01T04:22:29.290 INFO:teuthology.orchestra.run.smithi173.stderr:real 0m0.569s 2023-11-01T04:22:29.290 INFO:teuthology.orchestra.run.smithi173.stderr:user 0m1.602s 2023-11-01T04:22:29.290 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m0.059s 2023-11-01T04:22:29.320 INFO:teuthology.orchestra.run.smithi153.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-11-01T04:22:29.322 INFO:teuthology.orchestra.run.smithi153.stderr: 2023-11-01T04:22:29.322 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m0.603s 2023-11-01T04:22:29.322 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m1.808s 2023-11-01T04:22:29.322 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.116s 2023-11-01T04:22:29.323 INFO:tasks.ceph:Archiving logs... 2023-11-01T04:22:29.323 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312/remote/smithi153/log 2023-11-01T04:22:29.325 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-01T04:22:29.687 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312/remote/smithi173/log 2023-11-01T04:22:29.688 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-01T04:22:29.908 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312/remote/smithi181/log 2023-11-01T04:22:29.910 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-01T04:22:30.119 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312/remote/smithi190/log 2023-11-01T04:22:30.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-01T04:22:30.345 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-01T04:22:30.346 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-01T04:22:30.354 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2023-11-01T04:22:30.355 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-01T04:22:30.355 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-01T04:22:30.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-01T04:22:30.361 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-01T04:22:30.364 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-01T04:22:30.463 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-01T04:22:30.471 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-01T04:22:30.472 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-01T04:22:30.474 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-01T04:22:30.477 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-01T04:22:30.480 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-01T04:22:30.486 INFO:teuthology.orchestra.run.smithi153.stdout: remote refid st t when poll reach delay offset jitter 2023-11-01T04:22:30.486 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================== 2023-11-01T04:22:30.487 INFO:teuthology.orchestra.run.smithi153.stdout:*hv01.front.sepi 44.4.53.2 2 u 17 64 377 0.112 -1.304 0.684 2023-11-01T04:22:30.487 INFO:teuthology.orchestra.run.smithi153.stdout:+hv02.front.sepi 63.231.80.2 3 u 10 64 377 0.118 -0.573 0.703 2023-11-01T04:22:30.487 INFO:teuthology.orchestra.run.smithi153.stdout:+hv03.front.sepi 74.6.168.72 3 u 10 64 377 0.093 -1.449 0.963 2023-11-01T04:22:30.487 INFO:teuthology.orchestra.run.smithi153.stdout: hv04.front.sepi .INIT. 16 u - 128 0 0.000 +0.000 0.000 2023-11-01T04:22:30.489 INFO:teuthology.orchestra.run.smithi173.stdout: remote refid st t when poll reach delay offset jitter 2023-11-01T04:22:30.490 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================== 2023-11-01T04:22:30.490 INFO:teuthology.orchestra.run.smithi173.stdout:*hv01.front.sepi 44.4.53.2 2 u 12 64 377 0.126 -2.427 0.950 2023-11-01T04:22:30.490 INFO:teuthology.orchestra.run.smithi173.stdout:+hv02.front.sepi 63.231.80.2 3 u 4 64 377 0.091 -2.224 1.268 2023-11-01T04:22:30.490 INFO:teuthology.orchestra.run.smithi173.stdout:+hv03.front.sepi 74.6.168.72 3 u 12 64 377 0.095 -0.309 1.308 2023-11-01T04:22:30.490 INFO:teuthology.orchestra.run.smithi173.stdout: hv04.front.sepi .INIT. 16 u - 128 0 0.000 +0.000 0.000 2023-11-01T04:22:30.492 INFO:teuthology.orchestra.run.smithi181.stdout: remote refid st t when poll reach delay offset jitter 2023-11-01T04:22:30.492 INFO:teuthology.orchestra.run.smithi181.stdout:============================================================================== 2023-11-01T04:22:30.492 INFO:teuthology.orchestra.run.smithi181.stdout:*hv01.front.sepi 44.4.53.2 2 u 9 64 377 0.083 -1.946 0.638 2023-11-01T04:22:30.492 INFO:teuthology.orchestra.run.smithi181.stdout:+hv02.front.sepi 63.231.80.2 3 u 9 64 377 0.092 -1.496 0.915 2023-11-01T04:22:30.492 INFO:teuthology.orchestra.run.smithi181.stdout:+hv03.front.sepi 74.6.168.72 3 u 7 64 377 0.084 -0.469 0.416 2023-11-01T04:22:30.493 INFO:teuthology.orchestra.run.smithi181.stdout: hv04.front.sepi .INIT. 16 u - 128 0 0.000 +0.000 0.000 2023-11-01T04:22:30.493 INFO:teuthology.orchestra.run.smithi190.stdout: remote refid st t when poll reach delay offset jitter 2023-11-01T04:22:30.493 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================== 2023-11-01T04:22:30.494 INFO:teuthology.orchestra.run.smithi190.stdout:*hv01.front.sepi 44.4.53.2 2 u 9 64 377 0.069 -4.665 2.314 2023-11-01T04:22:30.494 INFO:teuthology.orchestra.run.smithi190.stdout:+hv02.front.sepi 63.231.80.2 3 u 11 64 377 0.060 -4.029 2.423 2023-11-01T04:22:30.494 INFO:teuthology.orchestra.run.smithi190.stdout:+hv03.front.sepi 74.6.168.72 3 u 10 64 377 0.094 -3.817 2.389 2023-11-01T04:22:30.494 INFO:teuthology.orchestra.run.smithi190.stdout: hv04.front.sepi .INIT. 16 u - 128 0 0.000 +0.000 0.000 2023-11-01T04:22:30.495 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-01T04:22:30.503 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-01T04:22:30.504 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-01T04:22:30.542 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-01T04:22:30.563 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-01T04:22:30.571 INFO:teuthology.task.internal:Duration was 1068.923440 seconds 2023-11-01T04:22:30.572 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-01T04:22:30.620 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-01T04:22:30.620 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-01T04:22:30.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-01T04:22:30.625 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-01T04:22:30.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-01T04:22:30.702 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-01T04:22:30.702 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2023-11-01T04:22:30.703 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-01T04:22:30.713 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2023-11-01T04:22:30.713 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-01T04:22:30.723 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2023-11-01T04:22:30.723 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-01T04:22:30.734 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2023-11-01T04:22:30.734 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-01T04:22:30.745 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-01T04:22:30.745 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-01T04:22:30.756 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-01T04:22:30.772 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-01T04:22:30.780 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-01T04:22:30.806 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-01T04:22:30.807 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-01T04:22:30.819 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-01T04:22:30.840 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-01T04:22:30.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-01T04:22:30.899 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-01T04:22:30.909 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-01T04:22:30.910 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-01T04:22:30.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-01T04:22:30.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-01T04:22:30.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-01T04:22:30.960 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-01T04:22:30.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-01T04:22:30.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-01T04:22:30.980 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-01T04:22:30.981 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2023-11-01T04:22:30.989 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2023-11-01T04:22:30.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-01T04:22:31.000 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2023-11-01T04:22:31.012 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2023-11-01T04:22:31.099 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-01T04:22:31.104 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi153.front.sepia.ceph.com, flagging run as failed 2023-11-01T04:22:31.105 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-01T04:22:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:22:31.111 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-01T04:22:31.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:22:31.116 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-01T04:22:31.150 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi190.front.sepia.ceph.com, flagging run as failed 2023-11-01T04:22:31.150 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-01T04:22:31.160 INFO:teuthology.task.internal:Transferring archived files... 2023-11-01T04:22:31.161 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312/remote/smithi153 2023-11-01T04:22:31.161 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-01T04:22:50.479 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2023-11-01T04:22:50.489 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312/remote/smithi153/coredump/1698812207.35781.core.gz: gzip compressed data, was "1698812207.35781.core", last modified: Wed Nov 1 04:22:31 2023, max compression, original size modulo 2^32 764674048 2023-11-01T04:22:50.490 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2023-11-01T04:22:53.641 INFO:teuthology.task.internal: core looks like: /tmp/tmp8rkdvlxf: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-osd -f --cluster ceph -i 12', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/bin/ceph-osd', platform: 'x86_64' 2023-11-01T04:22:53.710 INFO:teuthology.task.internal: dump_program: ceph-osd -f --cluster ceph -i 12 2023-11-01T04:22:53.711 DEBUG:teuthology.orchestra.run.smithi153:> which 'ceph-osd -f --cluster ceph -i 12' 2023-11-01T04:22:53.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:22:53.719 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 1: "which 'ceph-osd -f --cluster ceph -i 12'" 2023-11-01T04:22:53.719 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-01T04:22:53.732 INFO:teuthology.task.internal:Not uploading archives. 2023-11-01T04:22:53.733 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-01T04:22:53.753 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-01T04:22:53.753 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-01T04:22:53.764 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-01T04:22:53.767 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-01T04:22:53.768 INFO:teuthology.orchestra.run.smithi153.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 1 04:22 /home/ubuntu/cephtest 2023-11-01T04:22:53.768 INFO:teuthology.orchestra.run.smithi153.stdout: 658629 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Nov 1 04:04 /home/ubuntu/cephtest/archive 2023-11-01T04:22:53.768 INFO:teuthology.orchestra.run.smithi153.stdout: 658631 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 1 04:22 /home/ubuntu/cephtest/archive/syslog 2023-11-01T04:22:53.768 INFO:teuthology.orchestra.run.smithi153.stdout: 658633 4 -rw-rw-rw- 1 ubuntu ubuntu 823 Nov 1 04:22 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-11-01T04:22:53.769 INFO:teuthology.orchestra.run.smithi153.stdout: 658632 100 -rw-rw-r-- 1 ubuntu ubuntu 99439 Nov 1 04:22 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-01T04:22:53.769 INFO:teuthology.orchestra.run.smithi153.stdout: 663687 88 -rw-rw-rw- 1 ubuntu ubuntu 86332 Nov 1 04:22 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-11-01T04:22:53.769 INFO:teuthology.orchestra.run.smithi153.stdout: 658630 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 1 04:16 /home/ubuntu/cephtest/archive/coredump 2023-11-01T04:22:53.769 INFO:teuthology.orchestra.run.smithi153.stdout: 673808 64984 -rw------- 1 root root 764674048 Nov 1 04:16 /home/ubuntu/cephtest/archive/coredump/1698812207.35781.core 2023-11-01T04:22:53.770 INFO:teuthology.orchestra.run.smithi153.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-01T04:22:53.771 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-01T04:22:53.772 INFO:teuthology.orchestra.run.smithi173.stdout: 655476 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 1 04:22 /home/ubuntu/cephtest 2023-11-01T04:22:53.773 INFO:teuthology.orchestra.run.smithi173.stdout: 655477 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 1 04:22 /home/ubuntu/cephtest/archive 2023-11-01T04:22:53.773 INFO:teuthology.orchestra.run.smithi173.stdout: 658632 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 1 04:22 /home/ubuntu/cephtest/archive/syslog 2023-11-01T04:22:53.773 INFO:teuthology.orchestra.run.smithi173.stdout: 658634 4 -rw-rw-rw- 1 ubuntu ubuntu 829 Nov 1 04:22 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-11-01T04:22:53.773 INFO:teuthology.orchestra.run.smithi173.stdout: 658633 96 -rw-rw-r-- 1 ubuntu ubuntu 96249 Nov 1 04:22 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-01T04:22:53.773 INFO:teuthology.orchestra.run.smithi173.stdout: 663691 84 -rw-rw-rw- 1 ubuntu ubuntu 82033 Nov 1 04:22 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-11-01T04:22:53.774 INFO:teuthology.orchestra.run.smithi173.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-01T04:22:53.775 INFO:teuthology.orchestra.run.smithi181.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 1 04:22 /home/ubuntu/cephtest 2023-11-01T04:22:53.776 INFO:teuthology.orchestra.run.smithi181.stdout: 658634 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 1 04:22 /home/ubuntu/cephtest/archive 2023-11-01T04:22:53.776 INFO:teuthology.orchestra.run.smithi181.stdout: 658636 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 1 04:22 /home/ubuntu/cephtest/archive/syslog 2023-11-01T04:22:53.776 INFO:teuthology.orchestra.run.smithi181.stdout: 658638 4 -rw-rw-rw- 1 ubuntu ubuntu 822 Nov 1 04:22 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-11-01T04:22:53.776 INFO:teuthology.orchestra.run.smithi181.stdout: 658637 96 -rw-rw-r-- 1 ubuntu ubuntu 95909 Nov 1 04:22 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-01T04:22:53.776 INFO:teuthology.orchestra.run.smithi181.stdout: 663691 80 -rw-rw-rw- 1 ubuntu ubuntu 81647 Nov 1 04:22 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-11-01T04:22:53.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T04:22:53.778 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2792, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2782, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2545, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_420cd129397301b1521ceb74e1bdbd9b855d2189/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-11-01T04:22:53.779 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-01T04:22:53.789 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-01T04:22:53.839 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2023-11-01T04:22:53.840 INFO:teuthology.nuke:Checking targets against current locks 2023-11-01T04:22:53.842 INFO:teuthology.orchestra.run.smithi181.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-01T04:22:53.845 INFO:teuthology.orchestra.run.smithi190.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 1 04:22 /home/ubuntu/cephtest 2023-11-01T04:22:53.845 INFO:teuthology.orchestra.run.smithi190.stdout: 658630 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Nov 1 04:04 /home/ubuntu/cephtest/archive 2023-11-01T04:22:53.845 INFO:teuthology.orchestra.run.smithi190.stdout: 658632 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 1 04:22 /home/ubuntu/cephtest/archive/syslog 2023-11-01T04:22:53.846 INFO:teuthology.orchestra.run.smithi190.stdout: 658634 4 -rw-rw-rw- 1 ubuntu ubuntu 821 Nov 1 04:22 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-11-01T04:22:53.846 INFO:teuthology.orchestra.run.smithi190.stdout: 658633 96 -rw-rw-r-- 1 ubuntu ubuntu 96833 Nov 1 04:22 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-01T04:22:53.846 INFO:teuthology.orchestra.run.smithi190.stdout: 663688 84 -rw-rw-rw- 1 ubuntu ubuntu 82132 Nov 1 04:22 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-11-01T04:22:53.846 INFO:teuthology.orchestra.run.smithi190.stdout: 658631 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 1 04:16 /home/ubuntu/cephtest/archive/coredump 2023-11-01T04:22:53.846 INFO:teuthology.orchestra.run.smithi190.stdout: 673719 71380 -rw------- 1 root root 747888640 Nov 1 04:16 /home/ubuntu/cephtest/archive/coredump/1698812217.35957.core 2023-11-01T04:22:53.847 INFO:teuthology.orchestra.run.smithi190.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-01T04:22:53.872 DEBUG:teuthology.nuke:shortname: smithi153 2023-11-01T04:22:53.872 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-01T04:22:53.898 DEBUG:teuthology.nuke:shortname: smithi173 2023-11-01T04:22:53.898 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-01T04:22:53.903 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-01 03:56:01.288015', 'locked_by': 'scheduled_benhanokh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOAGLJhjDq8mvVkTG8Hw5tTgR0UZXjowuOsiYalT+QNbCXSyYSvs8LG7L/jZHCvB0sTKLuOqXRMsQl+LlBrIlOM='} 2023-11-01T04:22:53.924 DEBUG:teuthology.nuke:shortname: smithi181 2023-11-01T04:22:53.925 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-01T04:22:53.930 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-01 03:56:01.289629', 'locked_by': 'scheduled_benhanokh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKZi16MWKJndbPT3R1FdevTMb+dXFqXxHmJE/P6kQ+Gh9wecRp0CFuX/SllIWmHsP+6ZRYdXM51iHWKE79Ov4VI='} 2023-11-01T04:22:53.951 DEBUG:teuthology.nuke:shortname: smithi190 2023-11-01T04:22:53.952 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-01T04:22:53.960 INFO:teuthology.orchestra.console.smithi153:Power off 2023-11-01T04:22:53.961 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-01T04:22:53.972 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-01 03:56:01.293004', 'locked_by': 'scheduled_benhanokh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG6klwHOgaJO69D4LNOd3TcLmJkkJ3EdPRk+R27RAzf40b49HV2S9P8kZbO/WVyXSV3JNaS4kVTEr6Fr0WSh2BI='} 2023-11-01T04:22:53.992 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/benhanokh-2023-10-31_16:43:39-rados-DBG_maintain_prefix_itr_2-distro-default-smithi/7442312', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-01 03:56:01.291421', 'locked_by': 'scheduled_benhanokh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCRm8aIzpGl2fqDE99OAe/4/JqhCP3sSy1lyMbrSiFzkg5qUKVPKtmAiues2krHh9LOorVPXIHYKqqHJZs9ZqpM='} 2023-11-01T04:22:54.000 INFO:teuthology.orchestra.console.smithi173:Power off 2023-11-01T04:22:54.001 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-01T04:22:54.008 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2023-11-01T04:22:54.008 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-01T04:22:54.022 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2023-11-01T04:22:54.022 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-01T04:22:54.031 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2023-11-01T04:22:54.046 INFO:teuthology.orchestra.console.smithi181:Power off 2023-11-01T04:22:54.047 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-01T04:22:54.055 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2023-11-01T04:22:54.068 DEBUG:teuthology.orchestra.console.smithi181:power off output: Chassis Power Control: Down/Off 2023-11-01T04:22:54.068 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-01T04:22:54.082 INFO:teuthology.orchestra.console.smithi190:Power off 2023-11-01T04:22:54.082 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-01T04:22:54.092 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2023-11-01T04:22:54.106 DEBUG:teuthology.orchestra.console.smithi190:power off output: Chassis Power Control: Down/Off 2023-11-01T04:22:54.107 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-01T04:22:54.129 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2023-11-01T04:22:58.032 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-01T04:22:58.057 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-01T04:22:58.093 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-01T04:22:58.130 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-01T04:22:58.141 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2023-11-01T04:22:58.166 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2023-11-01T04:22:58.201 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2023-11-01T04:22:58.239 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2023-11-01T04:23:02.147 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-01T04:23:02.167 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-01T04:23:02.203 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-01T04:23:02.240 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-01T04:23:02.257 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2023-11-01T04:23:02.277 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2023-11-01T04:23:02.312 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is on 2023-11-01T04:23:02.350 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is off 2023-11-01T04:23:02.358 INFO:teuthology.orchestra.console.smithi153:Power off completed 2023-11-01T04:23:02.377 INFO:teuthology.orchestra.console.smithi173:Power off completed 2023-11-01T04:23:02.452 INFO:teuthology.orchestra.console.smithi190:Power off completed 2023-11-01T04:23:06.315 DEBUG:teuthology.orchestra.console.smithi181:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-01T04:23:06.425 DEBUG:teuthology.orchestra.console.smithi181:check power output: Chassis Power is off 2023-11-01T04:23:06.526 INFO:teuthology.orchestra.console.smithi181:Power off completed 2023-11-01T04:23:06.633 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} duration: 1068.9234399795532 failure_reason: 'Command failed on smithi153 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json''' flavor: default owner: scheduled_benhanokh@teuthology status: fail success: false 2023-11-01T04:23:06.634 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-01T04:23:06.715 INFO:teuthology.run:FAIL